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We  are  providing  this  final  report  for  review  and  comment.  We  considered 
management  comments  on  a  draft  of  this  report  when  preparing  the  final  report. 

DoD  Directive  7650.3  requires  that  all  recommendations  be  resolved  promptly.  As 
a  result  of  management  comments,  we  revised  Recommendation  3.  to  clarify  our  intent. 
We  request  that  the  Army  provide  additional  comments  on  Recommendations  2.  and  3. 
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Copies  of  the  management  comments  must  contain  the  actual  signature  of  the  authorizing 
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(wahopkins@dodig.osd.mil)  or  Mr.  Jim  Hutchinson  at  (703)  604-9060  (DSN  664-9060) 
(jhutchinson@dodig.osd.mil).  See  Appendix  C  for  the  report  distribution.  The  audit  team 
members  are  listed  inside  the  back  cover. 
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Report  No.  D-2002-103  June  14,  2002 

(Project  No.  D2000AS-0212.001) 

Certification  of  the  Reserve  Component  Automation  System 


Executive  Summary 


Who  Should  Read  This  Report  and  Why?  Managers  who  plan,  develop,  or  oversee 
DoD  automated  information  systems  will  be  interested  in  this  report. 

Background.  This  report  is  the  second  in  a  series  evaluating  certifications  that  DoD 
automated  information  systems  were  being  developed  in  accordance  with  the 
Clinger-Cohen  Act.  During  FYs  2000  and  2001,  Congress  required  that  the  Chief 
Information  Officer,  DoD,  before  approving  acquisition  Milestones  I,  II,  or  III  of  high- 
cost  information  systems,  evaluate  the  actions  taken  related  to  specific  requirements  of 
the  Clinger-Cohen  Act.  To  help  ensure  effective  oversight  of  DoD  information 
technology  investments,  Congress  included  Section  8121(b)  in  the  DoD  Appropriations 
Act  for  FY  2000,  which  also  required  the  Chief  Information  Officer,  DoD,  to  inform 
Congress  of  the  certifications  and  to  provide  confirmation  that  DoD  Components  took 
steps  to  meet  specific  requirements  of  the  Act.  The  Reserve  Component  Automation 
System  has  estimated  life  cycle  costs  of  $2.4  billion  for  FYs  1996  through  2007. 

Results.  The  Chief  Information  Officer,  DoD,  did  not  report  to  Congress  that 
development  of  the  Reserve  Component  Automation  System  did  not  fully  comply  with 
the  intent  of  the  Clinger-Cohen  Act.  The  limitations  directly  affected  three  of  the  five 
interest  items  that  were  specified  in  Section  8121(b)(2):  business  process 
reengineering,  analysis  of  alternatives,  and  performance  measures.  The  Chief 
Information  Officer  did  not  believe  the  weaknesses  for  business  process  reengineering 
and  analysis  of  alternatives  were  significant  enough  to  withhold  congressional 
certification  and  no  weaknesses  were  identified  for  functional  performance  measures 
even  though  none  were  specifically  developed.  Disclosure  of  compliance  limitations 
would  have  provided  Congress  with  a  more  accurate  measure  of  the  progress  and 
results  that  respective  information  technology  investments  made  in  complying  with  the 
Clinger-Cohen  Act.  To  meet  the  full  intent  of  the  Clinger-Cohen  Act,  the  application 
of  business  process  reengineering  and  analysis  of  alternatives  principles  should  be  used 
before  initiating  development  of  any  future  RCAS  increment  and  functional 
performance  measures  should  be  formally  established.  Additionally,  the  risks 
associated  with  exchanging  unencrypted  data  files  should  be  reevaluated.  Further,  out- 
year  funding  for  the  system  should  be  identified  and  related  congressional  reporting 
requirements  met.  See  the  finding  for  the  detailed  recommendations. 

Management  Comments  and  Audit  Response.  The  Chief  Information  Officer,  DoD, 
generally  concurred  with  the  audit  results  and  stated  that  he  would  develop  criteria  to 
enable  uniform  assessments  and  reporting  in  conjunction  with  the  Chief  Information 
Officers  of  the  DoD  Components.  The  Army  partially  concurred  with  the 
recommendations  on  reviewing  future  system  increments,  establishing  functional 
performance  measures,  evaluating  the  risks  associated  with  the  exchange  of 
unencrypted  data  files,  and  identifying  out-year  funding  for  the  Reserve  Component 
Automation  System.  Army  comments  on  the  recommendations  to  review  future  system 


increments  and  functional  performance  measures  were  not  clear;  therefore,  we  ask  for 
additional  comments.  We  also  request  the  Army  to  respond  to  the  recommendation 
concerning  congressional  reporting.  We  request  the  Army  to  provide  comments  on  the 
final  report  by  July  15,  2002.  See  the  Finding  section  of  the  report  for  a  discussion  of 
management  comments.  The  complete  text  of  management  comments  is  in  the 
Management  Comments  section. 
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Background 


In  the  mid-1990s.  Congress  passed  several  items  of  legislation  intended  to 
improve  the  management  and  performance  of  Federal  agencies.  The  reform 
legislation  responded  to  the  inability  of  Federal  agencies  to  effectively  manage 
the  development  and  production  of  information  technology  (IT)  systems  to  meet 
the  needs  of  functional  users.  One  major  reform  initiative  was  the  Information 
Technology  Management  Reform  Act  of  1996,  which  was  subsequently  retitled 
the  Clinger-Cohen  Act  of  1996  (CCA).  To  help  ensure  appropriate  management 
practices  in  developing  systems,  Congress  included  section  8121(b)  in  the  DoD 
Appropriations  Act,  FY  2000.  Section  8121(b)  required  the  Chief  Information 
Officer  (CIO),  DoD,  to  certify,  prior  to  approval  of  key  acquisition  review 
points  (milestones),  that  the  system  was  being  developed  in  accordance  with  the 
CCA.  Additionally,  the  CIO,  DoD,  was  required  to  notify  Congress  of  system 
certifications  and  confirm  the  performance  of  specific  “interest  items”  related  to 
CCA  tenets.  Appendix  B  provides  a  summary  of  the  CCA  and  the  specific 
section  8121(b)  requirements. 

DoD  Guidance.  The  specific  interest  items  iterated  in  section  8121(b)  were 
specifically  recognized  and  required  by  DoD  policy  and  guidance  before  the 
enactment  of  CCA  in  1996.  DoD  Directive  8000.1,  “Defense  Information 
Management  Program,”  October  27,  1992,  established  policy  and 
responsibilities  for  business  process  streamlining  and  improvements;  preparing 
and  validating  functional  economic  analyses,  which  include  analyses  of 
alternatives  and  investment  risk;  developing  functional  process  performance 
measures  and  assessments;  and  ensuring  appropriate  information  security.  In 
addition,  DoD  Directive  8120.1,  “Life-Cycle  Management  (LCM)  of 
Automated  Information  Systems  (AISs)  ,”  January  14,  1993,  1  emphasized  the 
importance  of  those  specific  section  8121(b)(2)  interest  items  that  are  critical  in 
the  “early-on”  IT  development  stages,  especially  those  related  to  improving 
business  processes  and  examining  alternatives  and  projecting  related  costs  and 
benefits. 

Acquisition  Program  Milestones.  A  milestone  is  a  decision  point  that 
separates  the  major  phases  of  an  acquisition  program.  DoD  acquisition  policy 
requires  a  milestone  decision  before  an  acquisition  program  may  progress  to  the 
next  acquisition  phase.  The  Assistant  Secretary  of  Defense  (Command, 

Control,  Communications,  and  Intelligence)  (ASD(C3I)),  as  the  Milestone 
Decision  Authority  for  major  automated  information  systems  approves  milestone 
decisions  for  high-cost  or  special  interest  IT  acquisition  programs. 

Prior  to  October  2000,  the  major  milestone  phases  included  Concept 
Exploration  (Phase  0),  Program  Definition  and  Risk  Reduction  (Phase  I), 
Engineering  and  Manufacturing  Development  (Phase  II),  and  Production, 
Fielding/Deployment,  and  Operational  Support  (Phase  III).  In  October  2000, 
DoD  substantially  revised  its  acquisition  guidance  and  requirements  to  reduce 


'DoD  Directive  5000.1,  “Defense  Acquisition,”  March  15,  1996,  canceled  DoD  Directive  8120.1  and 
incorporated  the  policies  and  requirements  on  life-cycle  management  for  automated  information  systems. 
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the  number  of  major  milestone  phases  and  their  associated  decision  points. 

DoD  also  revised  acquisition  regulations  to  clearly  and  effectively  implement 
various  aspects  of  IT  reform  legislation,  including  the  CCA.  Because  the 
Reserve  Component  Automation  System  (RCAS)  was  already  in  Milestone  III 
before  the  October  2000  change,  the  CIO  continued  to  use  the  existing  system  of 
milestone  designations  for  the  project. 

Reserve  Component  Automation  System.  In  1986,  the  RCAS  was  established 
to  provide  the  Army  National  Guard  and  the  Army  Reserve  with  a  single, 
extensive,  modern  automated  information  system  designed  to  support 
commanders,  staffs,  and  functional  managers  in  the  administration  and 
mobilization  of  the  Army  Reserve  Component.  The  mission  and  vision  of  the 
RCAS  are  to  support  daily  operations,  training,  and  administrative  tasks  for  all 
Guard  and  Reserve  echelons  and  to  provide  timely  and  more  accurate 
information  to  plan  and  support  mobilization.  When  it  is  fully  deployed,  the 
RCAS  will  link  more  than  10,500  Guard  and  Reserve  units  at  more  than 
4000  sites  located  in  all  50  states,  the  District  of  Columbia,  Guam,  Puerto  Rico, 
the  Virgin  Islands,  Europe,  and  the  Pacific  Rim. 

In  FY  1995,  the  Army  restructured  the  RCAS  project  to  constrain  cost  growth 
and  leverage  new  information  technology.  The  restructured  RCAS  project 
consisted  of  commercial  off-the-shelf  hardware  and  office  automation  software, 
government  off-the-shelf  software2,  and  newly  developed  software  applications 
that  were  integrated  into  a  personal  computer-based  architecture. 

RCAS  Increments.  The  1995  project  restructure  also  revised  the  RCAS 
acquisition  strategy  to  provide  an  incremental,  evolutionary  acquisition  approach 
that  included  development  and  deployment  of  capabilities  for  seven  increments. 
Early  project  increments  provided  the  necessary  infrastructure.  Increment  1, 
approved  in  September  1996  (Milestone  Ilia),  provided  commercial-off-the-shelf 
office  automation  software,  classified-capable  and  unclassified  workstations,  and 
wide  area  network  interconnectivity.  Increment  2,  approved  in  January  1998 
(Milestone  Illb),  introduced  data  servers  and  logistics  functional  software. 

Later  increments  focused  on  software  development  to  better  support  several 
functional  areas.  Increment  3,  approved  May  2000  (Milestone  IIIc),  provided 
force  authorization,  security,  and  training  functions.  Increment  4/53,  approved 
in  July  2001  (Milestone  Hid),  introduced  mobilization  planning  and  occupational 
health  management  and  added  additional  force  authorization  and  modernization, 
human  resources,  and  training  management  functionality.  Increment  6, 
scheduled  for  certification  for  compliance  with  the  requirements  of  the 
Clinger-Cohen  Act  during  FY  2002  (Milestone  Hie),  will  introduce 
mobilization  planning  and  occupational  health  management  functions.  Future 
increments  will  implement  user  requirements  in  the  order  of  priority  established 
by  the  Requirements  Control  Board  for  the  Reserve  Components.  RCAS  was 


2Software  previously  developed  to  military  or  Federal  specification  or  description  and  stocked  by  a 
distributor,  before  receiving  orders  or  contracts  for  its  sale. 

increments  4  and  5  were  combined  into  a  single  increment. 
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scheduled  to  be  fully  deployed  and  transitioned  to  a  separate  organization  within 
the  Reserve  Component  for  life-cycle  support  by  March  2003. 


The  estimated  life-cycle  costs  of  the  RCAS  project  for  Increments  1  through 
7  for  FYs  1996  through  2007  totaled  $2.4  billion.  Beyond  FY  2002,  all  costs 
will  be  user  costs  with  the  exception  of  activities  required  during  the  program 
transition  period  from  the  RCAS  Project  Management  Office  (PMO)  and  the 
contractor  to  the  users  and  the  RCAS  software  maintainer.  The  projected  return 
on  investment4  for  RCAS  Increment  3,  as  approved  in  May  2000,  was  4.5  to  1 
and  the  projected  return  on  investment  for  the  entire  RCAS  project  was  5  to  1 . 

The  ASD(C3I),  as  the  CIO,  DoD,  certified  on  March  28,  2000,  and  on  July  3, 
2001 5,  respectively,  that  Increment  3  and  Increments  4/5  of  the  Reserve 
Component  Automation  System  had  been  developed  in  accordance  with  the 
requirements  of  the  Clinger-Cohen  Act. 

Objectives 


The  audit  objective  was  to  determine  whether  DoD  oversight  processes  and 
procedures  provided  the  CIO,  DoD,  with  a  sufficient  basis  to  certify  that  the 
RCAS  was  being  managed  and  developed  in  accordance  with  the  CCA.  This 
report  is  the  second  of  a  series.  In  a  subsequent  report,  we  will  assess  DoD 
progress  in  implementing  the  CCA  and  review  related  management  controls.  A 
description  of  the  audit  scope  and  methodology  and  prior  coverage  related  to  the 
RCAS  project  is  shown  in  Appendix  A. 


4Retum  on  investment  is  the  ratio  of  the  present  value  of  benefits  to  the  present  value  of  costs. 

5Section  8121(b)  required  CIO,  DoD,  certification  during  FY  2000.  The  certification  requirement  was 
extended  through  FY  2001  by  Section  8102(b)  of  the  DoD  Appropriations  Act,  FY  2001. 
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Certification  of  the  Reserve  Component 
Automation  System  As  Compliant  with 
the  Clinger-Cohen  Act 

Limitations  of  the  RCAS  project  efforts  for  Increments  3  and  4/56  for 
compliance  with  the  intent  of  the  CCA  were  not  reported  by  the  CIO, 
DoD.  The  limitations  directly  affected  three  of  the  five  interest  items 
that  were  specified  in  Section  8121(b)(2).  This  condition  occurred 
because  the  CIO  did  not  believe  that  the  weaknesses  associated  with 
business  process  reengineering  and  the  analysis  of  alternatives  were 
significant  enough  to  withhold  the  CCA  certification.  In  addition,  the 
CIO  did  not  identify  any  weakness  in  performance  measures  even  though 
the  RCAS  PMO  and  the  Reserve  Components  had  not  established 
functional  performance  measures.  Accordingly,  Congress  was  not 
informed  that  the  RCAS  was  not  being  developed  in  full  compliance  with 
CCA  requirements. 

RCAS  Certification  Process 


The  RCAS  project  was  the  first  major  automated  information  system  in  DoD 
that  was  subject  to  the  Section  8121(b)  certification  process.  The  RCAS  was 
also  used  to  develop  a  template  and  a  certification  procedural  process  for  follow- 
on  projects.  The  PMO  prepared  a  compliance  report  for  the  RCAS  project, 
which  summarized  the  requirements  of  Section  8121(b),  provided  background 
information  on  the  RCAS  project,  and  outlined  the  actions  taken  by  project 
officials  on  the  five  interest  items  in  Section  8121(b)(2):  business  process 
reengineering,  analysis  of  alternatives,  economic  analysis,  performance 
measures,  and  information  assurance  strategy.  A  review  team,7  represented  by 
various  staff  offices  within  the  OSD,  then  prepared  the  congressional 
certification  report  for  signature,  stating  that  RCAS  Increment  3  was  being 
developed  in  accordance  with  the  CCA.  The  compliance  report  and  the  CIO 
certification  report  contained  essentially  the  same  information. 

On  February  25,  2000,  the  review  team  briefed  the  CIO  on  the  RCAS  draft 
certification  report  for  Congress.  The  briefing  included  confirmation  of  steps 
taken  by  the  PMO  to  address  each  of  the  five  specific  congressional  interest 
items. 

During  its  briefing  to  the  CIO,  the  review  team  presented  a  qualified  statement 
for  actions  regarding  business  processing  reengineering  and  the  analysis  of 


6Increments  4  and  5  were  combined  into  a  single  increment. 

7The  review  team  consisted  of  action  officers  from  the  Office  of  the  Secretary  of  Defense  (Command, 
Control,  Communications,  and  Intelligence);  Director,  Program  Analysis  and  Evaluation;  Director, 
Operational  Test  and  Evaluation;  and  Joint  Staff,  Director  for  Command,  Control,  Communications  and 
Computers  (J-6). 
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alternatives  because  business  process  work  flows  had  not  been  substantially 
redesigned.  Additionally,  the  PMO  had  not  considered  a  full  range  of 
approaches  for  the  analysis  of  alternatives  to  reduce  costs,  outsourcing  services, 
streamline  operations,  or  privatize  functions. 

Despite  those  weaknesses,  the  review  team  recommended  that  the  CIO  certify 
RCAS  Increment  3  as  CCA  compliant.  According  to  the  review  team,  the 
certification  report  could  not  qualify  or  restrict  the  level  of  steps  taken  by  the 
PMO  for  business  process  reengineering  and  analysis  of  alternatives;  a 
“qualified”  or  restricted  certification  was  not  an  option  because  a  project  either 
did  or  did  not  meet  the  CCA  certification  requirements.  The  CIO  tentatively 
approved  the  certification  during  the  briefing,  thus  authorizing  the  preparation 
of  the  official  certification  report  and  congressional  notification  letters  for 
submission  to  Congress. 

The  CIO  coordinated  the  certification  report  and  notification  letters  with,  and 
obtained  endorsement  by  the  Office  of  the  Under  Secretary  of  Defense 
(Comptroller);  the  Office  of  the  Assistant  Secretary  of  Defense  for  Legislative 
Affairs;  the  Office  of  the  Deputy  Under  Secretary  of  Defense  for  Program 
Integration;  the  Department  of  the  Army,  Office  of  the  Director  of  Information 
Systems  for  Command,  Control,  Communications,  and  Computers;  the  Office  of 
the  Director,  Program  Analysis  and  Evaluation  (PA&E);  the  Office  of  General 
Counsel;  the  National  Guard  Bureau,  Program  Executive  Office  for  Information 
Systems;  and  the  Chairman  of  the  RCAS  General  Officer  Steering  Committee. 
The  CIO  submitted  the  certification  report  to  Congress  on  March  28,  2000. 

We  focused  on  the  certification  of  Increment  3.  However,  because  the  CIO 
certified  and  submitted  the  certification  report  on  RCAS  Increments  4/5  to 
Congress  on  July  3,  2001,  while  the  audit  was  still  in  progress,  we  performed  a 
limited  review  of  the  certification  report  on  Increments  4/5.  We  determined 
that,  similar  to  the  certification  report  for  Increment  3,  the  CIO  did  not  report 
any  limitations  of  the  RCAS  project  efforts  for  compliance  with  the  intent  of 
CCA.  Unlike  Increment  3,  the  OSD  review  team  did  not  provide  a  formal 
documented  briefing  on  its  conclusions  regarding  certification  of  Increments  4/5 
(Milestone  Hid)  to  the  CIO.  According  to  the  staff  in  the  Office  of  the 
Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 
Intelligence)  (ASD[C3I])  staff,  Milestone  Illd  represented  a  recertification  of  the 
project’s  compliance  with  the  CCA  requirements  and  that,  with  the  exception  of 
determining  that  operations  and  support  for  RCAS  were  insufficiently  funded, 
and  updating  the  tables  and  exhibits  in  the  certification  report,  no  major  changes 
or  issues  occurred  after  the  OSD  review  team’s  assessment  of  Increment  3. 
Because  the  CIO  certified  Increment  4/5  based  on  similar  efforts  performed  by 
the  RCAS  PMO  and  assessments  made  by  the  OSD  review  team  on  Increment  3, 
we  concluded  that  the  concerns  presented  in  the  report  were  also  applicable  to 
Increment  4/5. 

Because  the  RCAS  was  the  first  system  certified  as  being  developed  in 
accordance  with  the  CCA,  we  believe  that  the  CIO  should  have  established  that 
two  classes  of  information  systems  are  subject  to  the  requirements  of  the  CCA 
within  DoD.  Specifically,  systems  that  started  development  or  were  in  an  early 
phase  of  development  after  the  enactment  of  the  CCA  in  1996  should  fully 
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comply  with  the  CCA.  However,  systems  such  as  RCAS,  which  were  in  an 
advanced  stage  of  development  and  deployment  when  the  CCA  was  enacted, 
most  likely  would  not  fully  meet  the  tenets  of  the  CCA  because  the  opportunities 
to  realize  the  most  substantial  benefits  from  “up  front”  efforts  such  as  business 
process  reengineering  or  analysis  of  alternatives  were  reduced  by  that  time. 

Business  Process  Reengineering 


Confirmation  of  Business  Process  Reengineering  Certification.  Although 
representatives  from  the  offices  of  the  ASD(C3I)  and  the  Director,  PA&E, 
acknowledged  that  DoD  and  the  Army  had  not  focused  on  the  use  of 
activity-based  costing  techniques  to  simplify  or  otherwise  redesign  business 
processes  before  investing  in  RCAS,  the  certification  report  to  Congress  did  not 
clearly  explain  that  RCAS  business  process  reengineering  efforts  did  not 
completely  meet  the  full  intent  of  the  CCA. 

RCAS  Business  Process  Reengineering  Efforts.  Although  the  RCAS 
investment  should  improve  and  support  work  processes  to  reduce  cost,  improve 
effectiveness,  and  implement  Government  and  commercial  off-the-shelf 
technology,  the  work  processes  of  the  Reserve  Components  were  not  fully 
subjected  to  business  process  reengineering. 

The  General  Accounting  Office,  “Business  Process  Reengineering  Assessment 
Guide,”  May  1997,  states  that  a  business  process  can  be  decomposed  into 
specific  activities,  measured,  modeled,  and  improved;  redesigned;  or 
eliminated.  Reengineering  identifies,  analyzes,  and  redesigns  an  organization’s 
core  business  processes  to  achieve  dramatic  improvements  in  critical 
performance  measures.  In  addition,  the  guide  states  that  dramatic  improvements 
realized  by  rethinking  how  the  organization’s  work  should  be  achieved, 
distinguishes  reengineering  from  process  improvement  that  focuses  on  functional 
or  incremental  improvement. 

Efforts  undertaken  by  the  RCAS  PMO  to  justify  information  technology 
investments  in  the  system  did  not  identify,  dramatically  redesign,  and  eliminate 
low  or  no  value-added  functions  or  work  processes  before  deciding  to  invest  in 
RCAS.  The  RCAS  PMO  indicated  that  such  efforts  were  not  a  top  priority  in 
1989  because  the  functional  users  were  focused  on  developing  requirements  and 
identifying  and  documenting  pre-automation  business  processes.  According  to 
the  PMO,  those  efforts  were  not  emphasized  when  OSD  reprogrammed  the 
Continental  Army  Management  Information  System  in  1986  as  RCAS  or 
restructured  the  RCAS  project  in  1995.  The  efforts  of  the  RCAS  stakeholders 
and  officials  related  to  business  process  reengineering  could,  at  best,  be 
considered  an  improvement  in  the  functional  process;  however,  those  efforts 
could  not  be  considered  a  redesign  and  reengineering  of  established  business 
processes  or  workflows. 

We  asked  the  PMO  to  provide  documentation  to  show  that  management 
considered  and  took  advantage  of  business  process  reengineering  opportunities 
before  making  a  commitment  and  commencing  the  development  or  acquisition  of 
software  applications  that  would  satisfy  the  functional  requirements  for  each 
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new  RCAS  increment.  The  PMO  stated  that  business  process  reengineering  was 
a  functional  community  responsibility  and  did  not  know  whether  the  Reserve 
Component  functional  communities  had  performed  independent  business  process 
reengineering  analyses  for  each  increment.  In  essence,  the  PMO,  in  conjunction 
with  the  functional  proponents,  had  not  performed  business  process 
reengineering  on  any  of  the  RCAS  increments  since  the  1996  enactment  of  the 
CCA. 

Staffs  of  the  ASD(C3I)  and  PMO  also  indicated  that  business  process 
reengineering  was  not  a  primary  consideration  when  the  Army  reprogrammed 
the  Continental  Army  Management  Information  System  and  renamed  it  as  RCAS 
in  1986  because  the  significance  of  the  derived  advantages  of  business  process 
reengineering  were  not  widely  recognized  and  emphasized  at  that  time; 
however,  when  the  Army  restructured  the  RCAS  project  in  FY  1995,  DoD 
policy  required  DoD  Components  to  consider  business  process  streamlining. 

The  PMO  exerted  extensive  efforts  to  overcome  the  inadequacies  of  existing 
methods  and  procedures  by  proposing  to  automate  inefficient,  functionally 
disconnected,  and  manual  processes.  The  PMO  estimated  that  about 
$3.5  billion  (94  percent)  of  the  benefits  derived  from  the  RCAS  included 
productivity  improvements  that  would  result  from  automating  work  processes 
rather  than  from  the  functional  reengineering  or  redesign  of  those  processes. 
Although  automation  of  work  processes  would  require  fewer  Reserve 
Component  personnel  to  perform  administrative  tasks,  there  was  no  expectation 
to  reduce  the  number  of  personnel.  Instead,  the  Reserve  Components  planned  to 
use  the  extra  time  to  provide  additional  training  for  personnel. 

Analysis  of  Alternatives  and  Economic  Analysis 


Policy.  DoD  Instruction  7041.3,  “Economic  Analysis  for  Decision  Making,” 
November  1995,  contains  policy  for  economic  analysis  and  analysis  of 
alternatives.  An  analysis  of  alternatives  and  an  economic  analysis  are  directly 
related;  effective  use  of  an  analysis  of  alternatives,  in  conjunction  with  an 
economic  analysis,  provides  a  viable  basis  for  evaluating  potential  solutions  and 
selecting  the  most  cost-effective  alternative.  The  analysis  of  alternatives 
generally  starts  with  a  broad  base  of  possible  solutions  to  meet  a  mission  need. 
When  the  field  of  possible  solutions  is  narrowed  to  a  few  realistic  alternatives, 
the  principles  of  economic  analysis  and  its  tools  of  cost-benefit  analysis  and 
return-on-investment  are  applied  to  identify  the  most  promising  solution. 

Analysis  of  Alternatives.  The  PA&E  office  qualified  its  assessment  of  the 
analysis  of  alternatives  in  the  February  2000  briefing  to  the  CIO  because  the 
August  1996  analysis  of  alternatives  did  not  consider  a  full  range  of  alternatives 
to  reduce  cost,  such  as  outsourcing  specific  functions  or  streamlining  or 
privatizing  routine  administrative  processes.  Routine  administrative  processes 
include  personnel  activities,  payroll,  training,  and  human  resources. 
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According  to  DoD  Instruction  7041.3,  each  feasible  alternative  for  meeting  an 
objective  must  be  considered  and  its  life-cycle  costs  and  benefits  should  be 
evaluated.  Alternatives  dismissed  as  “infeasible”  must  be  discussed,  but  need 
not  be  formally  compared  in  the  economic  analysis. 

The  PMO  stated  that  the  RCAS  was  exempt  from  outsourcing  routine 
administrative  processes  because  the  system  was  established  under  title  10, 
United  States  Code  of  the  Armed  Services  Program  and  because  it  supported 
numerous  inherent  Government  functions,  such  as  manning,  equipping,  training, 
and  sustaining  the  Army’s  Reserve  Components. 

We  acknowledge  that  the  mobilization  capability  of  the  RCAS  may  be  an 
inherent  Government  function  but  believe  that  the  routine  administrative 
processes  of  RCAS  are  not.  The  Office  of  Management  and  Budget 
Circular  A-76,  “Performance  of  Commercial  Activities,”  August  4,  1983, 
(Revised  1999)  states  that  an  inherent  Government  function  is  a  function  that  is 
so  intimately  related  to  the  public  interest  as  to  mandate  performance  by 
Government  employees.  The  PMO  did  not  provide  any  documentation  to  show 
how  RCAS  was  an  inherent  Government  function.  Additionally,  the  PMO  did 
not  consider,  in  the  analysis  of  alternatives,  the  opportunity  to  competitively 
source  the  day-to-day,  repetitive  administrative  tasks  and  work  processes  of  the 
project  and  did  not  discuss  the  infeasibility  of  that  option. 

Economic  Analysis.  We  examined  the  related  Milestone  III  economic  analysis, 
dated  August  1996,  and  identified  no  major  deficiencies  based  on  the 
requirements  of  DoD  Instruction  7041.3.  Also,  neither  the  PMO  nor  the 
ASD(C3I)  presented  any  economic  analysis  issues  to  the  CIO  during  the  RCAS 
certification  briefing.  However,  we  noted  that  the  benefits  used  in  the 
computation  of  the  return  on  investment  consisted  of  “soft  dollars”  or  benefits 
that  could  not  be  quantitatively  tracked  through  the  budget  process.  The  PA&E 
office  questioned  the  amount  of  actual  benefits  because  benefits  were  primarily 
based  on  productivity  gains.  Because  the  return  on  investment  was  based  on 
increased  productivity,  the  use  of  performance  measures  to  assess  the  functional 
benefits  of  the  RCAS  investment  becomes  even  more  important. 

Performance  Measures 


Functional  proponents  of  RCAS  did  not  establish  a  performance  measurement 
plan  to  assess  functional  performance  or  to  identify  whether  the  desired  results 
were  being  achieved  after  the  deployment  of  RCAS.  Specifically,  new 
processes  were  not  compared  against  measures  of  outcome,  output,  and 
efficiency  of  RCAS  in  order  to  continually  monitor  performance  and  make 
further  refinements.  In  addition,  the  PMO  did  not  use  benchmarks  to  assess  the 
efficiency  of  work  process  improvements. 

Functional  Performance  Measures.  According  to  the  PMO,  proponents  did 
not  establish  functional  performance  measures  because  those  measures  were  not 
considered  a  top  priority  in  1989  when  the  functional  users  focused  on 
identifying  and  documenting  pre-automation  business  processes  and  translating 
the  results  into  requirements  for  RCAS.  In  addition,  user  representatives, 
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including  members  of  the  customer  focus  team  for  RCAS,  stated  that  they  did 
not  establish  functional  performance  measures.  The  CCA  requires  that 
performance  measurements  be  prescribed  for  information  technology  acquired 
for  or  used  by  the  executive  agency.  The  performance  measurements  should 
indicate  how  well  the  information  technology  supports  projects  of  the  executive 
agency. 

Although  functional  proponents  established  key  performance  measures  for  the 
system’s  operational  performance,  including  operational  effectiveness  and 
suitability,  cost,  timeliness,  and  quality,  the  measurements,  when  implemented, 
did  not  measure  the  outcome  of  the  investment  in  or  functional  benefits  of 
RCAS.  Without  functional  performance  measures,  the  Army  and  the  Reserve 
Components  would  not  be  able  to  determine,  quantitatively,  how  well 
RC AS-improved  processes  met  mission  goals,  or  identified  problems  in  meeting 
those  goals.  An  example  of  a  functional  performance  measure  may  be  to  show 
how  many  soldiers  would  be  relieved  of  administrative  workloads  in  order  to 
engage  in  more  training  because  of  RCAS  automation. 

Benchmarks.  Although  the  CIO  certification  report  stated  that  benchmarks 
were  used  to  derive  risk-adjusted  alternative  technical  solutions  during  the  1995 
project  restructure,  the  PMO  did  not  provide  any  documentation  to  support  its 
benchmarking  efforts.  For  benchmarking,  the  CCA  requires  that,  where 
comparable  processes  and  organizations  in  the  public  or  private  sectors  exist, 
process  performance  should  be  quantitatively  benchmarked  against  such 
processes  in  terms  of  cost,  speed,  productivity,  and  quality  of  outputs  and 
outcomes. 

The  General  Accounting  Office,  “Business  Process  Reengineering  Assessment 
Guide,”  dated  May  1997,  indicated  that  benchmarks  are  instrumental  in 
identifying  gaps  between  an  organization’s  process  performance  and  that  of 
leading  organizations  and  in  understanding  how  those  leaders  have  changed  their 
structures,  work  processes,  and  lines  of  business  to  improve  performance 
dramatically. 

According  to  the  RCAS  PMO,  no  documented  evidence  was  available  to  show 
that  either  the  functional  users  or  the  PMO  used  benchmarks  for  existing  work 
processes  with  internal  or  commercial  organizations.  Unless  an  organization 
uses  benchmarks  to  measure  its  process  performance  with  the  goals  and 
performance  of  leading  organizations,  it  is  difficult  to  establish  reference  points 
for  setting  meaningful  improvement  goals.  Benchmarks,  when  used  in 
conjunction  with  performance  measurement,  present  a  sound  method  to  establish 
a  credible  business  case  for  changing  work  processes  of  an  organization. 

Information  Assurance 


Although  the  certification  testing  of  RCAS  met  the  requirements  of  DoD 
Instruction  5200.40,  “DoD  Information  Technology  Security  Certification  and 
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Accreditation  Process,”  December  30,  1997,  and  the  congressional  certification 
requirements  for  the  CCA,  the  PMO  did  not  use  encryption8  techniques  to 
safeguard  sensitive  but  unclassified  data. 

In  January  1997,  at  the  request  of  the  RCAS  Program  Executive  Office,  the 
Director  of  Information  Systems  for  Command,  Control,  Communications,  and 
Computers,  who  was  the  Designated  Approving  Authority  for  the  RCAS, 
granted  a  deferment  for  deploying  data  encryption  hardware  devices,  pending 
final  determination  of  a  software  encryption  or  common  hardware  solution.  As 
of  January  2002,  the  deferment  was  still  in  effect. 

We  requested  and  the  PMO  provided  a  listing  of  28  current  and  projected 
system  interfaces.  Of  the  28  interfaces  with  other  systems,  16  data  exchanges 
used  File  Transfer  Protocol,  which  is  a  service  that  supports  file  transfers 
between  local  and  remote  computers,  including  the  Internet.  No  documentation 
was  available  to  indicate  that  RCAS  encrypted  any  of  its  data.  The  PMO  stated 
that  electronic  external  interfaces  were  not  authenticated9  or  encrypted  and  that 
engineering  efforts  were  ongoing  with  the  owners  of  the  data  to  provide  security 
during  electronic  transmission.  Though  the  data  were  unclassified,  they  may 
have  contained  sensitive  information,  such  as  personnel  data,  manpower 
allocation,  and  force  structure.  If  the  data  remain  unencrypted,  system  users 
may  be  vulnerable  to  network  attacks  or  compromise,  such  as  eavesdropping 
and  playback. 

The  Director  of  Information  Systems  for  Command,  Control,  Communications, 
and  Computers  and  the  PMO  should  review  the  risks  associated  with  passing 
unsecured  sensitive  data  and  implement  encryption  technology,  such  as  the 
Secure  Socket  Layer,  if  appropriate,  to  reduce  the  risk  of  inadvertent  disclosure 
of  sensitive  but  unclassified  data. 

Oversight 


Although  the  CIO,  senior  DoD  officials,  and  action  officers  reviewed  key 
project  documents,  such  as  the  acquisition  strategy,  the  Operations  Concept 
document  (the  RCAS  project’s  operational  requirements  document),  the  Test 
and  Evaluation  Master  Plan,  and  the  Acquisition  Program  Baseline,  the  CIO 
needs  to  establish  uniform  criteria  to  determine  compliance  with  the  CCA.  The 
criteria  should  include  the  need  for  the  Overarching  and  Integrating  Integrated 
Product  Teams10  to  improve  their  involvement  in  the  certification  process. 


Encryption  is  the  transformation  of  data  into  a  form  unreadable  by  anyone  without  a  secret  decryption 
key  and  ensures  privacy  by  keeping  the  information  hidden  from  anyone  for  whom  it  was  not  intended. 

Authentication  is  the  process  of  determining  the  identity  of  a  user  attempting  to  access  a  system. 

10In  1995,  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  and  the  ASD(C3I) 
issued  guidance  entitled  “Rules  of  the  Road,  A  Guide  for  Leading  Successful  Integrated  Product  Teams” 
that  emphasized  the  importance  and  advantages  of  minutes  of  meetings,  what  they  should  include,  and 
that  all  members  of  the  IPT  should  be  provided  final  minutes  within  2  working  days  after  the  deadline 
for  the  receipt  of  comments. 
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Additionally,  the  CIO  and  senior  DoD  officials  need  to  make  sure  that  the  Army 
complies  with  congressional  direction  regarding  the  absence  of  funding  for 
sustainment  operations  and  support  for  the  RCAS  project. 

CIO,  DoD.  The  CIO,  DoD,  certified  that  Increment  3  of  the  RCAS  project  was 
developed  in  accordance  with  the  CCA,  although  the  basis  for  the  certification 
was  confusing  because  the  CIO  had  not  established  universal  criteria  or  a 
consistent  approach  to  determine  the  adequacy  of  compliance  with  the  CCA 
requirements.  The  first  report  in  this  series,  Inspector  General,  DoD,  Report 
No.  D-2001-137,  “Certification  of  the  Defense  Civilian  Personnel  Data 
System,”  June  7,  2001,  recommended  that  the  CIO  clarify  and  enhance  the 
criteria  and  approach  to  be  used  by  DoD  Components  for  determining  whether 
major  automated  information  systems  are  developed  in  accordance  with  the 
CCA.  Therefore,  this  report  will  not  include  a  recommendation  addressing  the 
matter. 

Overall,  the  CIO  could  improve  oversight  responsibilities  through  periodic 
verification  of  information  provided.  Because  CIO  staff  members  seldom 
performed  detailed  reviews  of  project  documentation,  we  concluded  that  prudent 
verification  could  substantially  improve  the  effectiveness  of  oversight 
responsibilities.  This  report  will  not  include  a  recommendation  addressing  this 
matter  because  Inspector  General,  DoD,  Report  No.  D-2000-137  recommended 
that  the  CIO  strengthen  oversight  processes,  including  the  process  for  certifying 
that  major  automated  information  systems  are  developed  in  accordance  with  the 
CCA,  by  periodically  confirming  the  accuracy  and  adequacy  of  information 
reported  by  DoD  Components. 

Army  CIO.  Absent  compliance  criteria  from  the  CIO,  DoD,  the  Army  CIO 
established  a  checklist,  which  included  Section  8121(b)  requirements,  to  assess 
compliance  with  the  CCA  In  December  1999,  the  Army  CIO  approved  the 
CCA  compliance  of  Milestone  IIIc,  Increment  3,  based  on  the  PMO  self- 
assessment  checklist  submission  that  addressed  the  five  interest  items  outlined  in 
the  OSD(C3I)  guidance. 

Information  Technology  Overarching  Integrated  Product  Team.  The 

Information  Technology  Overarching  Integrated  Product  Team  (Overarching 
IPT)  was  minimally  involved  in  the  oversight  of  the  RCAS.  The  primary  role  of 
the  Overarching  IPT  was  to  provide  advice  to  the  CIO  during  milestone  reviews. 
The  Director,  C3I  Acquisition  (now  the  Director,  Investment  and  Acquisition), 
Office  of  the  Deputy  Assistant  Secretary  of  Defense  (C3I)  CIO,  chaired  the 
Overarching  IPT  that  was  composed  of  senior  managers  representing  the  primary 
staff  assistants  with  an  interest  in  the  RCAS.  The  Overarching  IPT  included 
senior  managers  from  the  offices  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics;  the  Under  Secretary  of  Defense 
(Comptroller);  the  Director,  Operational  Test  and  Evaluation;  the  Director, 
Program  Analysis  and  Evaluation;  the  Director,  Defense  Information  Systems 
Agency;  the  Director,  Command,  Control,  Communications,  and  Computer 
Systems,  Joint  Staff;  and  user  representatives. 
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Although  the  Overarching  IPT  reviewed  and  concurred  with  draft  acquisition 
decision  memorandums  before  formal  RCAS  milestone  decisions,  it  did  not  meet 
during  milestone  reviews  to  discuss  the  progress  and  status  of  the  RCAS  project 
and  did  not  help  identify  potential  programmatic  problems  .  Instead,  the 
Overarching  IPT  relied  on  a  lower-level.  Integrating  IPT  to  provide  critical 
RCAS  oversight  review  and  direction. 

Integrating  Integrated  Product  Team.  The  Integrating  IPT  members  indicated 
that  they  continuously  monitored  the  RCAS  project;  however,  they  were  unable 
to  provide  summaries  or  minutes  of  meetings  or  any  memorandums  for  the 
record  on  the  level  of  input  and  guidance  by  representatives  on  significant 
programmatic  issues  discussed  and  resolved  during  reviews. 

The  Integrating  IPT  was  co-chaired  by  the  RCAS  Project  Manager  and  action 
officers  from  the  offices  of  the  ASD(C3I);  the  Director,  PA&E;  the  Director, 
Operational  Test  and  Evaluation;  and  the  Joint  Staff.  From  September  1996 
through  March  2000,  the  Integrating  IPT  met  13  times  to  monitor  program 
status,  testing  strategy,  software  encryption,  information  assurance,  training, 
incremental  fielding  and  testing  issues,  and  Section  8121(b)  certification.  The 
Integrating  IPT  also  tracked  action  items,  audits,  reviews,  and  corrective  actions 
to  address  deficiencies  identified  by  the  Inspector  General,  DoD,  and  the 
General  Accounting  Office. 

During  the  audit,  the  Integrating  IPT  showed  improvements  in  maintaining 
informative  minutes  of  IPT  meetings.  For  the  April  2001  review  of  RCAS 
Milestone  Hid  (Increment  4/5),  the  Integrating  IPT  produced  a  memorandum  for 
the  record  that  disclosed  the  specific  issues  discussed,  actions  needed  to  address 
those  issues,  and  the  next  proposed  Integrating  IPT  milestone  review. 

Program  Funding.  Although  it  had  planned  to  fully  deploy  RCAS  by  the  end 
of  FY  2002  (later  changed  to  March  2003),  the  Army  still  had  not  provided 
funding  for  operations  and  support  requirements  for  the  system.  In  the  House 
of  Representatives  Armed  Services  Committee  (the  Committee)  Report 
No.  106-616,  “National  Defense  Authorization  Act  for  FY  2001,”  May  12, 
2000,  the  Committee  expressed  a  concern  that,  without  continued  support  and 
modernization,  the  Army  Reserve  could  experience  a  serious  deterioration  in 
readiness. 

The  Committee  also  expressed  concern  that  the  Army  had  allocated  only  limited 
funding  for  the  RCAS  project  in  the  Future  Years  Defense  Program.  In  order  to 
ensure  that  the  program  continued  to  enable  the  effective  administrative  support 
and  mobilization  capability  required  by  the  Reserve  Components,  the  Committee 
expected  the  Department  of  the  Army  to  program  sufficient  funds  for  RCAS. 

The  Committee  directed  the  Secretary  of  Defense  to  provide  a  report  no  later 
than  March  1,  2001,  to  the  Senate  and  House  Committees  on  Armed  Services 
detailing  programmed  funds  for  RCAS  for  FYs  2002  through  2007.  As  of 
June  2002,  the  Army  had  not  completed  and  submitted  the  report  to  the 
Committees. 

In  the  January  2001  Defense  Acquisition  Executive  Summary  report,  the  RCAS 
PMO  reported  a  $765  million  unfunded  requirement  in  FYs  2002  through  2007 
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for  operations  and  support  required  to  operate  and  maintain  the  RCAS 
infrastructure.  During  its  Milestone  Hid  review  of  Increment  4/5  in  April  2001, 
the  Integrating  IPT  decided  that  the  unfunded  requirement  should  be 
acknowledged  in  the  CIO  certification  compliance  package  and  in  the 
Milestone  IHd  Acquisition  Decision  Memorandum.  Specifically,  in  the  July  2, 
2001,  Milestone  IHd  Acquisition  Decision  Memorandum,  the  Overarching  IPT 
tasked  the  Army  and  the  RCAS  PMO  to  jointly  work  towards  a  strategy  to 
resolve  the  unfunded  requirement.  The  Army  and  the  RCAS  PMO  were  to 
report  their  findings  and  recommendations  to  the  Overarching  IPT  and  the 
RCAS  General  Officer  Steering  Committee  before  the  end  of  FY  2001.  In 
addition,  the  Acquisition  Decision  Memorandum  stated  that  the  RCAS  “Other 
Procurement”  funding  for  FY  2002  would  not  be  obligated  until  the  CIO,  DoD, 
reviewed  and  approved  the  study  and  its  recommendations. 

Conclusion 


The  CIO  certified  that  RCAS  was  developed  in  accordance  with  the  CCA, 
although  business  process  reengineering,  an  analysis  of  alternatives,  and 
performance  measures  were  not  fully  compliant  with  the  intent  of  CCA 
requirements.  Milestone  III  was  too  late  in  the  RCAS  development  process  to 
effectively  use  and  fully  capitalize  upon  these  investment  tools.  The 
certification  report  to  Congress  should  have  explained  that,  due  to  RCAS 
Milestone  III  decision  point,  the  project  was  not  fully  subjected  to  steps  that 
could  justify  more  than  a  qualified  confirmation.  Disclosure  of  compliance 
limitations  would  have  provided  a  more  accurate  measure  of  the  progress  and 
results  that  respective  IT  investments  made  in  complying  with  the  CCA. 

Also,  although  RCAS  was  past  the  stage  where  business  processes  reengineering 
and  an  analysis  of  alternatives  could  be  most  useful,  performance  measures  to 
measure  the  functional  benefits  of  RCAS  after  full  deployment  of  the  system 
should  still  be  established  and  would  still  be  beneficial.  Further,  the  application 
of  business  process  reengineering  and  analysis  of  alternatives  principles  would 
still  be  useful  prior  to  initiating  development  of  any  future  RCAS  increment. 

Management  Comments  on  the  Finding  and  Audit  Response 


Management  Comments.  The  Deputy  Chief  Information  Officer,  DoD, 
concurred  and  stated  that  although  certain  CCA  compliance  limitations  were 
recognized  by  DoD  officials  at  the  time  of  certification,  achieving  full 
compliance  was  also  recognized  as  not  practical  because  RCAS  development 
was  too  advanced  to  remedy  weaknesses  that  occurred  early  in  the  development 
process. 

On  behalf  of  the  National  Guard  Bureau  and  the  RCAS  PMO,  the  National 
Guard  Chief  Information  Officer  and  Program  Executive  Officer  for  Information 
Systems  provided  consolidated  comments  that  nonconcurred  with  the  finding. 
The  National  Guard  Chief  Information  Officer  stated  that  activities  related  to 
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each  of  the  interest  items  were  completed  before  the  CCA  was  enacted  and  that 
the  efforts  followed  the  regulations,  guidance,  and  best  practices  that  were 
available  at  the  time. 


Audit  Response.  We  agree  that  achieving  full  compliance  may  not  have  been 
practical  because  RCAS  was  already  at  Milestone  III  and  that  efforts  followed 
the  guidance  that  existed  at  the  time.  The  review  team  also  recognized 
limitations  to  full  compliance.  Accordingly,  the  CIO,  DoD,  certification  that 
RCAS  was  being  developed  in  accordance  with  the  CCA  should  have  been 
appropriately  qualified. 

The  National  Guard  Chief  Information  Officer  disagreed  with  other  aspects  of 
the  finding  and  discussion.  A  summary  of  additional  management  comments 
and  the  audit  response  is  in  Appendix  C.  The  full  text  of  all  management 
comments  is  in  the  Management  Comments  section  of  this  report. 

Recommendations,  Management  Comments  and  Audit 
Response 


Revised,  Deleted,  and  Renumbered  Recommendations.  Based  on 
management  comments,  we  revised  draft  Recommendation  3. a.  to  better  express 
our  intent.  Based  on  management  comments  and  additional  audit  work,  we 
deleted  draft  Recommendation  4. a.  Additionally,  for  clarity,  we  converted  draft 
Recommendations  3. a.  and  3.b.  into  distinctly  separate  recommendations. 
Therefore,  draft  Recommendations  3. a.  and  3.b.  have  been  renumbered  as 
Recommendations  3.  and  4.,  respectively.  We  also  renumbered  draft 
Recommendations  4.b.  and  5.  to  Recommendations  5.  and  6.,  respectively. 

1.  We  recommend  that  the  Chief  Information  Officer,  DoD,  establish  policy 
to  report  limitations  of  project  efforts  for  full  compliance  with  the  intent  of 
the  Clinger-Cohen  Act  requirements. 

Management  Comments.  The  Deputy  Chief  Information  Officer,  DoD, 
concurred  and  cited  continuing  efforts  to  develop,  in  coordination  with  DoD 
Component  Chief  Information  Officers,  specific  criteria  to  enable  uniform 
assessments  of  Clinger-Cohen  Act  compliance.  Recent  efforts  include  the 
development,  during  2001,  of  an  updated  Clinger-Cohen  Act  certification  and 
confirmation  template.  Further,  two  web  sites  were  developed  to  enhance  the 
procedures  and  approach  used  by  DoD  Components  for  determining 
Clinger-Cohen  Act  compliance. 

Audit  Response.  Although  Clinger-Cohen  Act  compliance  reporting  to 
Congress  is  no  longer  required,  DoD  acquisition  guidance  continues  to  require, 
prior  to  project  initiation  or  milestone  approval  by  the  Milestone  Decision 
Authority,  confirmation  by  DoD  Component  CIOs  that  mission-critical  or 
mission-essential  information  systems  are  being  developed  in  accordance  with 
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the  Clinger-Cohen  Act.  Accordingly,  the  development  of  specific  criteria 
should  help  to  obtain  more  consistent  and  uniform  assessments  of  Clinger-Cohen 
Act  compliance. 

2.  We  recommend  that  the  Chief,  National  Guard  Bureau;  the  Chief,  Army 
Reserve;  and  the  Reserve  Component  Automation  System  Management 
Officer  review  the  five  Section  8121(b)(2)  interest  items  for  the  proposed 
capabilities  of  Increment  6,  as  appropriate,  and  any  future  increments,  and 
determine  whether  the  selected  solution  complies  with  the  intent  of  the 
requirements  of  the  Clinger-Cohen  Act. 

Management  Comments.  The  National  Guard  and  Army  Reserve  both 
provided  qualified  concurrences.  The  National  Guard  stated  that  business 
process  reengineering  and  analysis  of  alternatives  were  accomplished  during  the 
earlier  phases  of  RCAS  development  and  that  the  reviews  performed  for  each 
RCAS  increment  validate  earlier  milestone  decisions  by  ensuring  that 
requirements  are  satisfied.  The  Army  Reserve  stated  that  it  strives  to  apply  the 
principles  of  the  Clinger-Cohen  Act  and  to  manage  its  network  and  associated 
systems  and  applications  on  an  enterprise- wide  basis.  Accordingly,  actions 
taken  for  RCAS  will  also  be  the  actions  taken  on  behalf  of  all  Army  Reserve 
systems. 

Audit  Response.  Management  comments  did  not  address  the  intent  of  the 
recommendation.  We  recognize  that  business  process  reengineering  and 
analysis  of  alternative  efforts  were  performed  more  than  7  years  ago  and  prior 
to  the  RCAS  restructure  in  1995.  We  also  recognize  that  information 
technology  has  changed  substantially  over  those  7  years.  For  information 
technology  systems  that  are  incrementally  developed  over  a  period  of  time, 
business  processes  or  technological  alternatives  for  implementing  those 
processes  should  be  periodically  reexamined,  as  intended  by  the  Clinger-Cohen 
Act.  Accordingly,  we  request  that  the  Chief,  National  Guard  Bureau  and  the 
Chief,  Army  Reserve  reconsider  their  responses  to  the  recommendation  and 
provide  additional  comments. 

3.  We  recommend  that  the  Chief,  National  Guard  Bureau  and  the  Chief, 
Army  Reserve  require  functional  proponents  of  the  Reserve  Component 
Automation  System  to  establish  functional  performance  measures  to  better 
assess  both  the  initial  and  future  impact  of  RCAS  on  supported 
functionalities. 

Management  Comments.  The  National  Guard  concurred  in  principle.  The 
National  Guard  agreed  on  the  importance  of  performance  measures  and  stated 
that  functional  performance  measures  relating  to  administration,  interoperability, 
logistics,  and  security  already  exist  and  are  documented.  The  existing 
performance  measures  quantitatively  set  standards  for  hundreds  of  Reserve 
Component  processes  and  compare  attributes  of  the  new  RCAS  business 
processes  to  the  pre-RCAS  business  process. 

The  Army  Reserve  concurred  with  the  intent  of  the  recommendation,  but  stated 
that  because  RCAS  supported  only  some  of  the  business  processes  within  each 
of  the  functional  areas,  measurement  of  RCAS  operations  in  isolation  of  other 


15 


supporting  systems  would  potentially  be  counterproductive  or  misleading. 
Alternatively,  the  Army  Reserve  is  pursuing  a  more  holistic  means  to  catalog 
and  measure  supporting  information  systems  by  building  an  information 
technology  portfolio,  which  will  identify  all  systems  and  applications  that 
contribute  to  functional  mission  accomplishment.  As  a  contributor  to  several 
functional  areas,  RCAS  is  captured  in  the  Army  Reserve’s  information 
technology  portfolio. 

Audit  Response.  Management  comments  were  partially  responsive.  Although 
the  National  Guard  indicated  that  functional  performance  measures  had  been 
developed  and  were  in  place,  the  system  of  measures  cited  focused  on  enabling 
economic  analyses  and  operational  testing,  rather  than  on  measuring  and 
assessing  key  improvements  in  each  of  the  1 1  functional  areas  supported  by 
RCAS.  Some  of  the  existing  measures  could  be  used  to  assess  RCAS  functional 
performance  and  to  help  track  improvements  from  future  RCAS  enhancements. 
However,  it  is  not  clear  how  the  functional  contributions  provided  by  RCAS  are 
assessed  and  monitored  as  part  of  the  Army  Reserve  information  technology 
portfolio.  Accordingly,  we  revised  the  recommendation  to  clarify  our  intent  and 
request  that  the  National  Guard  and  the  Army  Reserve  provide  additional 
comments  on  the  revised  recommendation. 

4.  We  recommend  that  the  Chief,  National  Guard  Bureau  and  the  Chief, 
Army  Reserve  assess  the  risk  of  exchanging  unencrypted  files  containing 
sensitive  data  between  the  Reserve  Component  Automated  System  for 
proposed  and  fielded  increments  and  other  networked  systems  and,  if 
appropriate,  implement  encryption  technology. 

Management  Comments.  The  National  Guard  and  Army  Reserve  conditionally 
concurred.  The  National  Guard  stated  that  the  risks  associated  with  exchanging 
unencrypted  fdes  had  been  previously  assessed  by  the  Army  and  was 
categorized  as  low.  Additionally,  the  Designated  Approval  Authority  for  RCAS 
will  reexamine  the  subject  risks  during  the  next  accreditation  review,  which  is 
scheduled  for  November  2002.  Accordingly,  the  National  Guard  felt  that  the 
recommended  action  by  the  National  Guard  and  the  Army  Reserve  is  not 
required.  The  Army  Reserve  stated  that  RCAS  interfaces  with  other  Army 
Reserve  systems  within  a  secure  network  boundary,  which  minimizes  the 
security  risks.  The  Army  Reserve  also  cited  initiatives  to  consolidate  the  RCAS 
infrastructure  into  consolidated  data  centers  with  tightly  controlled  access  in  and 
out  of  those  centers. 

Although  not  required  to  respond,  the  Army  Deputy  Chief  Information  Officer 
(The  Army  Deputy  CIO)  concurred.  Citing  his  responsibilities  as  the 
Designated  Approving  Authority  for  RCAS,  he  stated  that  the  risks  related  to 
the  exchange  of  unencrypted  files  will  be  specifically  examined  during  the 
scheduled  reaccreditation  of  RCAS  in  November  2002.  Further,  the  Army 
Deputy  CIO  will  specifically  review  and  determine  whether  RCAS  file 
exchanges  should  continue  to  be  unencrypted  or  additional  security  measures  are 
merited. 
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5.  We  recommend  that  the  Reserve  Component  Automation  System  Project 
Manager  develop  a  plan,  prior  to  the  Milestone  Ille  review,  for  the 
approval  of  the  Chief  Information  Officer,  Department  of  the  Army,  that 
funds  the  operation  and  support  of  the  Reserve  Component  Automation 
System  for  FYs  2002  through  2007. 

Management  Comments.  The  National  Guard,  responding  for  the  RCAS 
Project  Manager,  conditionally  concurred.  The  RCAS  Project  Manager  and  the 
Army  CIO  worked  jointly  to  identify  sufficient  funding  to  sustain  RCAS. 
Additionally,  the  Army  recently  directed  more  than  $300  million  to  address 
RCAS  life-cycle  shortfalls  in  funding.  As  a  result,  the  Army  Deputy  Chief  of 
Staff  for  Program  Analysis  and  Evaluation  declared  RCAS  as  affordable. 

Although  not  required  to  respond,  the  Army  Deputy  CIO  concurred  and  stated 
that  sufficient  funding  to  sustain  RCAS  for  FYs  2002  through  2007  had  been 
identified.  Accordingly,  the  Army  CIO  certified  on  February  25,  2002,  to  the 
CIO,  DoD,  that  RCAS  out-year  funding  issues  had  been  resolved.  Because 
RCAS  funding  issues  were  resolved,  the  CIO,  DoD,  approved  Milestone  Ille 
and  authorized  the  fielding  of  RCAS  Increment  6  on  March  25,  2002. 

6.  We  recommend  that  the  Assistant  Secretary  of  the  Army,  Acquisition, 
Logistics  and  Technology,  expedite  a  report,  which  was  due  by  March  1, 
2001,  to  the  Senate  and  House  Committees  on  Armed  Services,  that  details 
sufficient  programmed  funds  for  administrative  support  and  mobilization 
capability  for  RCAS  for  FYs  2002  through  2007. 

Management  Comments  Required.  The  Assistant  Secretary  did  not  comment 
on  a  draft  of  this  report.  We  ask  that  the  Assistant  Secretary  provide  comments 
on  the  final  report. 
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Appendix  A.  Audit  Process 

Scope  and  Methodology 


We  evaluated  the  basis  for  the  certification  made  to  Congress  in  response  to 
Section  8121(b),  and  the  effectiveness  of  oversight  provided  by  the  Overarching 
IPT,  the  Acquisition  Oversight  IPT,  and  the  milestone  reviews.  Specifically, 
we  reviewed  the  certification  process,  including  the  compliance  report  prepared 
by  the  RCAS  PMO,  briefing  charts  used  to  brief  the  Deputy  CIO  on  the  RCAS 
certification  process,  and  the  certification  report  submitted  to  Congress  by  the 
CIO. 

We  discussed  various  aspects  of  the  RCAS  certification  process,  procedures, 
and  information  provided  to  Congress  with  staffs  of  the  Director,  Army 
National  Guard;  the  Director,  Program  Analysis  and  Evaluation;  and  the  CIO. 

We  inquired  about  the  oversight  provided  by  the  OSD  Information  Technology 
Overarching  IPT.  We  reviewed  project  documents  dating  from  July  1987 
through  July  2001.  We  determined  whether  project  officials  adequately 
prepared  key  documentation  prior  to  the  Milestone  IIIc  review  on  December  14, 
1999. 

We  reviewed  the  Acquisition  Decision  Memorandums  issued  for  the 
Milestone  Ilia  (September  1996),  Illb  (January  1998),  and  IIIc  (May  2000) 
reviews  and  determined  whether  the  exit  criteria  provided  in  the  Acquisition 
Decision  Memorandums  were  well-defined  and  enforced  by  the  Milestone 
Decision  Authority  and  his  staff.  Finally,  we  reviewed  the  actions  taken  in 
response  to  prior  audits  and  reviews  of  the  RCAS  project. 

General  Accounting  Office  High-Risk  Area.  The  General  Accounting  Office 
has  identified  several  high-risk  areas  in  DoD.  This  report  provides  coverage  of 
the  Information  Management  and  Technology  high-risk  area. 

Use  of  Computer-Processed  Data.  We  did  not  use  computer-processed  data  to 
perform  this  audit. 

Use  of  Technical  Assistance.  We  received  technical  assistance  from  a 
computer  engineer  in  the  Technical  Assessment  Division,  Audit  Followup  and 
Technical  Support  Directorate.  The  computer  engineer  reviewed  RCAS 
documentation  on  information  security  and  testing.  Specifically,  the  computer 
engineer  reviewed  the  system  security  authorization  agreement,  the  certification 
report,  the  risk  assessment,  the  security  user’s  manual,  the  security  standing 
operating  procedures  guide,  and  the  system  security  architecture. 

Audit  Type,  Dates,  and  Standards.  We  performed  this  audit  from  January 
2001  through  April  2002,  in  accordance  with  generally  accepted  government 
auditing  standards. 
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Contacts  During  the  Audit.  We  visited  or  contacted  individuals  and 
organizations  within  DoD.  Further  details  are  available  upon  request. 

Prior  Coverage 


Inspector  General  of  the  Department  of  Defense  (IG  DoD) 

IG  DoD  Audit  Report  No.  97-019,  “Evaluation  of  the  Reserve  Component 
Automation  System,”  November  1,  1996 

IG  DoD  Audit  Report  No.  D-2000-137,  “Certification  of  the  Defense  Civilian 
Personnel  Data  System,”  June  7,  2001 
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Appendix  B.  Clinger-Cohen  Act  of  1996  and 

Statutory  Requirements 


Clinger-Cohen  Act  of  1996.  The  CCA  requires  Federal  agencies  to  focus  on 
the  results  achieved  through  IT  investments  while  streamlining  the  IT 
procurement  process.  Specifically,  the  CCA  introduced  additional  precision  and 
structure  into  the  way  that  agencies  approach  the  selection,  acquisition,  and 
management  of  IT.  A  primary  requirement  of  the  CCA  was  to  establish  the 
position  of  the  CIO  for  each  Federal  agency. 

Therefore,  in  June  1997,  the  Secretary  of  Defense  designated  the  ASD(C3I)  as 
the  CIO  for  DoD  and  conferred  the  authority  and  responsibility  for 
implementing  certain  aspects  of  the  CCA.  The  CIO  responsibilities  include: 

•  designing  and  implementing  a  process  for  maximizing  the  value  and 
assessing  and  managing  the  risks  of  DoD  IT  acquisitions; 

•  institutionalizing  performance-  and  results-based  IT  management;  and 

•  providing  advice  and  other  assistance  to  the  Secretary  of  Defense  and 
other  senior  DoD  managers  to  ensure  that  IT  acquisition  and 
information  resources  are  managed  in  accordance  with  the  policies  of 
the  CCA. 

The  CIO  is  also  responsible  for  the  management  and  oversight  of  all  DoD  IT 
systems.  Specific  responsibilities  include  overseeing  the  performance  of  IT 
projects  and  measuring  project  progress  through  system  milestone  reviews. 

Statutory  Requirements.  Congress  enacted  Section  8121(b)  ,  “Certifications 
as  to  Compliance  with  the  Clinger-Cohen  Act”  of  the  DoD  Appropriations  Act 
for  FY  2000,  which  states: 

(1)  During  the  fiscal  year  2000,  a  major  automated  information  system 
may  not  receive  Milestone  I  approval,  Milestone  II  approval,  or 
Milestone  III  approval  within  the  Department  of  Defense  until  the 
Chief  Information  Officer  certifies,  with  respect  to  that  milestone,  that 
the  system  is  being  developed  in  accordance  with  the  Clinger-Cohen 
Act  of  1996  (40  U.S.C  1401  et  seq.).  The  Chief  Information  Officer 
may  require  additional  certifications,  as  appropriate,  with  respect  to 
any  such  system. 

(2)  The  Chief  Information  Officer  shall  provide  the  congressional 
defense  committees  timely  notification  of  certifications  under 
paragraph  (1).  Each  such  notification  shall  include,  at  a  minimum,  the 
funding  baseline  and  milestone  schedule  for  each  system  covered  by 
such  a  certification  and  confirmation  that  the  following  steps  have 
been  taken  with  respect  to  the  system: 
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A)  Business  process  reengineering. 

B)  An  analysis  of  alternatives. 

C)  An  economic  analysis  that  includes  a  calculation  of  the 
return  on  investment. 

D)  Performance  measures. 

E)  An  information  assurance  strategy  consistent  with  DoD 
Command,  Control,  Communications,  Computers, 

Intelligence,  surveillance,  and  Reconnaissance 
Architecture  Framework. 

Section  8121(b)  requirements  were  only  applicable  during  FY  2000.  However, 
Congress  extended  essentially  the  same  certification  requirements  through 
FY  2001  by  enacting  Section  8102(b),  “Certifications  as  to  Compliance  with  the 
Clinger-Cohen  Act,”  of  the  DoD  Appropriations  Act  for  FY  2001.  The  DoD 
Appropriations  Act  for  FY  2002,  section  8104(b)  again  extended  a  certification 
requirement  prior  to  milestone  approval,  but  limited  the  scope  of  that 
requirement  to  major  automated  information  systems  for  financial  management 
and  required  that  the  Under  Secretary  of  Defense  (Comptroller)  certify  that  the 
system  is  being  developed  in  accordance  with  the  DoD  Financial  Management 
Modernization  Plan. 
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Appendix  C.  Summary  of  Management 

Comments  on  the  Finding  and 
Audit  Response 


Responding  jointly  for  the  Chief,  National  Guard  Bureau  and  the  RCAS  PMO, 
the  National  Guard  Bureau  Chief  Information  Officer  and  Program  Executive 
Officer  for  Information  Systems  disagreed  with  several  topical  area  discussions 
in  the  draft  report.  Management  nonconcurred  with  our  discussions  of  the 
RCAS  certification  process,  business  process  reengineering,  analysis  of 
alternatives,  economic  analysis,  performance  measures,  the  integrating  IPT, 
milestone  exit  criteria,  and  the  conclusion.  Management  also  commented  on 
several  inaccuracies  in  the  draft  report. 

Management  Comments  on  the  RCAS  Certification  Process.  Regarding  the 
qualifications  for  business  process  reengineering  and  analysis  of  alternatives 
presented  by  the  OSD  review  team  in  its  briefing  of  the  draft  CIO  RCAS 
certification  report,  management  indicated  that  the  qualifications  were  not 
merited.  Management  stated  that  the  RCAS  PMO  provided  an  extensive  set  of 
artifacts  regarding  the  occurrence  of  and  content  of  those  activities. 

Audit  Response.  As  discussed  in  the  report,  the  OSD  review  team  did  not 
consider  steps  taken  by  the  RCAS  project  sufficiently  rigorous  to  meet  the  intent 
of  the  CCA  for  business  process  reengineering  or  analysis  of  alternatives. 

Management  Comments  on  Business  Process  Reengineering.  Management 
stated  that  the  CCA  requires  agencies  to  revise  mission-related  processes  and 
that  the  RCAS  PMO  provided  a  variety  of  documentation  showing  that  business 
processes  were  refined  prior  to  significant  RCAS  investment.  Additionally,  the 
CCA  makes  no  mention  of  business  process  reengineering  or  specific  techniques 
to  use  for  process  revision.  Citing  functional  process  improvement  that  began 
in  1989  and  continued  through  the  RCAS  restructure  in  1995,  management  cited 
extensive  study  and  documentation  to  create  and  refine  core  business  processes 
across  11  functional  areas.  By  1996,  RCAS  had  completed  business  process 
reengineering  efforts  and  set  requirements  for  all  increments.  Accordingly,  it 
would  be  unreasonable  to  expect  the  RCAS  PMO  or  the  functional  users  to 
conduct  further  business  process  reengineering  on  any  of  the  increments, 
regardless  of  their  occurring  after  the  enactment  of  the  CCA. 

Audit  Response.  We  agree  with  management  on  the  stated  requirements  of  the 
CCA.  However,  Section  8121(b)  required  that  the  CIO,  in  the  certification 
report  to  Congress,  describe  steps  taken  in  regard  to  business  process 
reengineering.  The  differences  between  business  process  reengineering  and 
business  process  improvements  are  discussed  in  the  report.  The  report 
recognized  business  process  improvements  undertaken  during  RCAS 
development,  but  also  it  concluded  that  those  efforts  do  not  meet  the  higher 
standards  inherent  in  business  process  reengineering  or  the  intent  of  the  CCA. 
We  do  not  necessarily  agree  with  management  that  once  requirements  are 
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established,  it  is  unreasonable  to  conduct  further  business  process  reengineering. 
Such  an  approach  may  preclude  leveraging  subsequent  technological  or 
functional  improvements. 

Management  Comments  on  Analysis  of  Alternatives.  For  the  range  of 
alternatives  considered  during  the  August  1996  analysis,  management  stated  that 
the  analysis  leading  to  the  RCAS  restructure  in  1995  demonstrated  that  multiple 
technical  solutions,  project  organizations,  and  day-to-day  business  processes 
were  evaluated  before  selecting  the  RCAS  solution.  Regarding  the  possible 
outsourcing  of  selected  RCAS  functions,  management  stated  that,  in  meeting  the 
Federal  Activities  Inventory  Reform  Act  requirement  for  agencies  to  annually 
identify  those  activities  not  inherently  governmental,  senior  Army  executives 
have  not  identified  any  jobs  or  functions  that  RCAS  supports.  Further,  the 
National  Guard  outsources  to  the  States  those  functions  not  uniquely  military  or 
inherently  governmental. 

Audit  Response.  We  acknowledge  that  the  cited  alternatives  were  analyzed 
prior  to  the  RCAS  restructure  in  1995.  However,  the  RCAS  PMO  provided  no 
evidence  that  the  analyses  of  alternatives,  including  those  documented  for  the 
Milestone  Ilia  review  in  August  1996,  considered  the  privatization  of  routine 
administrative  processes.  Accordingly,  we  concluded  that  RCAS  managers  did 
not  meet  one  of  the  tenets  of  the  CCA:  determining  whether  the  function  could 
be  performed  more  effectively  and  at  less  cost  by  the  private  sector.  We  also 
acknowledge  that  the  thrust  of  the  Federal  Activities  Inventory  Reform  Act  is  to 
reduce  the  federal  workforce  by  outsourcing  those  positions  and  activities  that 
are  not  inherently  governmental.  However,  that  Act  was  not  passed  until 
October  1998,  well  past  the  period  discussed.  Because  privatization  of  routine 
administrative  processes  was  not  addressed  by  RCAS  or  functional  officials  in 
the  analysis  of  alternatives,  we  continue  to  conclude  that  the  RCAS  certification 
should  have  been  appropriately  qualified. 

Management  Comments  on  Economic  Analysis.  Regarding  the  quantification 
and  tracking  of  RCAS  benefits,  management  stated  that  RCAS  management  has 
quantified  the  productivity  improvements  derived  by  the  project  by  evaluating 
labor  requirements,  cycle  time,  frequency,  and  output  quality  for  RCAS 
processes.  The  RCAS  PMO  also  performs  post-implementation  reviews  to 
quantitatively  track  the  actual  realization  of  cost  savings  and  productivity 
improvements. 

Audit  Response.  We  agree  that  RCAS  quantitatively  expressed  cost  and 
benefits  in  the  formal  cost  benefit  analysis  produced  to  support  each  increment. 
We  also  agree  that  the  productivity  improvements  for  each  increment  are 
assessed  and  quantitatively  expressed  during  each  post-implementation  review. 
However,  that  was  not  the  intended  focus  of  our  discussion  in  the  draft  report. 
Our  intent  was  to  describe  that  the  anticipated  benefits  of  RCAS  were  primarily 
based  on  productivity  improvements  and  not  on  actual  cost  reductions  or  “hard” 
savings.  We  have  amended  the  economic  analysis  discussion  on  page  8  of  this 
final  report  to  better  express  our  intent. 
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Management  Comments  on  Performance  Measures.  Management  stated  that 
key  functional  requirements  were  established  during  the  RCAS  restructure  in 
1995.  Grouped  into  six  performance  measurement  categories,  these 
requirements  were  then  incorporated  into  the  Acquisition  Program  Baseline. 

Key  performance  parameters  and  measures  of  effectiveness  were  then 
designated,  including  many  functional  performance  measures  such  as 
mobilization  order  processing  times  and  maintenance  response  times.  To 
determine  how  well  RCAS  meets  the  key  performance  parameters,  the  Army 
Test  and  Evaluation  Command  employs  both  operational  and  mobilization 
activities  to  measure  functional  performance.  Additionally,  the  functional 
performance  of  each  RCAS  increment  is  independently  analyzed.  As  of  March 
2002,  management  stated  that  more  than  100  functional  processes  had  been 
measured.  This  approach  provides  the  RCAS  Project  Manager  and  the  user 
community  with  both  functional  and  system  performance  measures. 

As  to  the  use  of  benchmarking,  management  stated  that  the  RCAS  functional 
communities  used  benchmarking  during  the  process  selection  workshops 
conducted  during  the  late  1980s.  Additionally,  benchmarking  was  used,  among 
other  techniques,  during  the  RCAS  restructure  to  derive  low,  medium,  and 
high-risk  alternative  sustainment  strategies.  A  ’’Benchmark  Interview  Guide” 
was  used  to  evaluate  4  existing  Government  programs  and  at  least  15 
commercial  vendors. 

Audit  Response.  We  agree  that  some  of  the  measures  established  could  be  used 
as  effective  functional  performance  measures.  However,  the  system  of 
measures  cited  by  management  was  established  primarily  to  measure  system 
performance,  to  establish  performance  parameters  for  operational  testing,  and  to 
aid  in  determining  the  systems  economic  benefits.  As  expressed  in  the  report, 
the  main  purpose  of  functional  performance  measures  is  to  enable  the  functional 
community,  or  communities,  to  quantitatively  assess  the  amount  of  functional 
gain  provided  from  its  investment  in  a  new  system.  After  the  system  is 
completed,  ongoing  measurement  of  functional  performance  should  also  enable 
the  ftinctional  community  to  continually  assess  whether  investments  in  system 
maintenance  or  upgrades  are  worthwhile  from  a  functional  perspective. 
Additionally,  continual  measurement  of  functional  performance  provides  a 
performance  baseline  from  which  the  functional  gains  attributable  to  future  new 
systems  can  be  soundly  determined. 

As  to  benchmarking,  management  did  not  provide  supporting  documentation 
regarding  the  use  of  benchmarking  in  the  late  1980s.  Further,  management 
statements  regarding  the  use  of  benchmarking  to  evaluate  alternative  sustainment 
strategies  could  be  misleading  because  they  used  benchmarking  to  select  an 
approach  to  system  maintenance,  which  is  not  relevant  to  functional 
performance  measures.  However,  because  the  functional  communities 
supported  by  RCAS  did  not  establish  a  functional  performance  baseline  prior  to 
RCAS  development,  benchmarking  could  be  of  benefit  in  the  establishment  of 
functional  performance  measures.  Benchmarks  representing  the  functional 
performance  of  leading  organizations  could  be  used  as  a  functional  performance 
goal.  A  system  of  functional  performance  measures  would  enable  RCAS  users 
to  measure  progress  toward  achieving  that  goal. 
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Management  Comments  on  Integrating  Integrated  Product  Team.  As  to  the 

date  of  the  Milestone  Hie  review,  management  stated  that  the  review  occurred  in 
October  2001,  but  that,  as  of  January  2002,  the  related  Acquisition  Decision 
Memorandum  had  not  been  issued. 

Audit  Response.  We  agree  with  management  and  have  accordingly  revised  the 
report. 

Management  Comments  on  Milestone  Exit  Criteria.  In  regards  to  the 
effectiveness  of  RCAS  training,  management  stated  that  the  exit  criteria 
established  by  the  DoD  CIO  for  Milestone  IIIc  (Increment  3)  were  met.  More 
important,  management  stated  that  substantial  improvement  had  been  made  in 
training  RCAS  users  since  the  fielding  of  Increment  3.  Those  improvements  are 
illustrated  in  the  operational  testing  reports  of  the  Army  Test  and  Evaluation 
Command  for  Increments  4/5  and  6.  Management  stated  that  during  the  recent 
evaluation  of  RCAS  Increment  6,  the  Army  Test  and  Evaluation  Command  rated 
training  as  one  of  the  project’s  areas  of  strength. 

Audit  Response.  As  a  result  of  management  comments,  we  reviewed  the 
operational  test  report  for  Increment  4/5 .  Because  the  operational  test  report 
was  not  yet  available,  we  also  reviewed  the  Army  Test  and  Evaluation 
Command  briefing  charts  for  Increment  6  and  discussed  the  adequacy  of  RCAS 
Increment  6  training  with  Army  Test  and  Evaluation  Command  personnel.  As  a 
result,  we  agree  that  RCAS  user  training  had  significantly  improved  since 
Increment  3  and  was  no  longer  a  reportable  weakness.  Accordingly,  we 
removed  the  subject  discussion  and  associated  recommendation  from  this  final 
report. 

Management  Comments  on  Conclusion.  Regarding  the  application  of  CCA 
principles  to  future  RCAS  increments  that  contain  new  functions,  management 
stated  that  no  new  functions  have  entered  the  RCAS  production  process  since 
the  project’s  Milestone  III  decision  in  1996. 

Audit  Response.  Our  intent  was  that  RCAS  managers  reexamine  business 
process  reengineering  and  analysis  of  alternatives  prior  to  initiating  development 
of  any  future  increment.  Although  the  RCAS  Milestone  III  was  approved  in 
1996,  the  dynamics  of  the  IT  marketplace  continue  to  provide  opportunities  for 
enabling  business  process  reengineering  efforts.  Accordingly,  investments  in 
future  RCAS  increments  should  be  examined  within  the  context  of  the 
Clinger-Cohen  Act.  We  revised  the  conclusion  in  this  report  to  better  express 
our  intent. 

Management  Comments  on  Inaccuracies  in  the  Draft  Report.  Management 
identified  items  requiring  correction,  such  as:  RCAS  is  a  project  instead  of  a 
program;  the  head  of  the  RCAS  PMO  is  the  project  manager  and  not  the  project 
management  officer;  and  the  Reserve  Components  should  be  referred  to  as  the 
Army  Reserve  Component. 

Audit  Response.  We  made  those  corrections. 
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Appendix  D.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
Under  Secretary  of  Defense  (Comptroller) 

Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 

Director,  Program  Analysis  and  Evaluation 

Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and  Intelligence) 
Deputy  Assistant  Secretary  of  Defense  (Deputy  Chief  Information  Officer) 

Director,  Investment  and  Acquisition 


Department  of  the  Army 

Assistant  Secretary  of  the  Army  (Acquisition,  Logistics,  and  Technology) 

Inspector  General,  Department  of  the  Army 

Auditor  General,  Department  of  the  Army 

Chief  Information  Officer 

Chief,  National  Guard  Bureau 

Program  Executive  Officer  for  Information  Systems 

Project  Manager,  Reserve  Component  Automation  System 
Chief,  Army  Reserve 


Department  of  the  Navy 

Naval  Inspector  General 

Auditor  General,  Department  of  the  Navy 


Department  of  the  Air  Force 

Auditor  General,  Department  of  the  Air  Force 


Other  Defense  Organizations 

Director,  Defense  Contract  Audit  Agency 
Director,  Defense  Contract  Management  Agency 
Director,  Defense  Finance  and  Accounting  Service 
Director,  Defense  Logistics  Agency 
Director,  National  Security  Agency 

Inspector  General,  National  Security  Agency 
Inspector  General,  Defense  Intelligence  Agency 
Inspector  General,  Defense  Threat  Reduction  Agency 
Commandant,  Defense  Systems  Management  College 
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Non-Defense  Federal  Organization 

Office  of  Management  and  Budget 
National  Security  Division 
Office  of  Information  and  Regulatory  Affairs 


Congressional  Committees  and  Subcommittees,  Chairman  and 
Ranking  Minority  Member 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Committee  on  Government  Reform 

House  Subcommittee  on  Government  Efficiency,  Financial  Management,  and 
Intergovernmental  Relations,  Committee  on  Government  Reform 
House  Subcommittee  on  National  Security,  Veterans  Affairs,  and  International 
Relations,  Committee  on  Government  Reform 
House  Subcommittee  on  Technology  and  Procurement  Policy,  Committee  on 
Government  Reform 
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Assistant  Secretary  of  Defense  (Command, 
Control,  Communications,  and  Intelligence) 
Comments 


DEPARTMENT  OF  DEFENSE 

6000  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-6000 

March  20,  2002 

CHIEF  INFORMATION  OFFICER 

MEMORANDUM  FOR  DIRECTOR,  ACQUSITION  MANAGEMENT 

DoD  OFFICE  OF  THE  INSPECTOR  GENERAL 

SUBJECT:  DoDIG  Audit,  Certification  of  the  Reserve  Component  Automation  System 
(RCAS)  (Project  No.  D2000AS-02 12.001) 

•  FINDING:  Limitations  of  the  RCAS  Program  efforts  for  Increments  3  and  4/5  for 
compliance  with  the  intent  of  the  CCA  were  not  reported  by  the  CIO,  DoD.  The  limitations 
directly  affected  three  of  the  five  interest  items  that  were  specified  in  Section  8121(b)(2).  This 
condition  occurred  because  the  CIO  did  not  believe  that  the  weaknesses  associated  with 
business  process  reengineering  and  the  analysis  of  alternatives  were  significant  enough  to 
withhold  the  CCA  certification.  In  addition,  the  CIO  did  not  identify  any  weakness  in 
performance  measures  even  though  the  RCAS  PMO  and  the  Reserve  Components  had  not 
established  functional  performance  measures.  Accordingly,  Congress  was  not  informed  that 
the  RCAS  was  not  being  developed  in  full  compliance  with  CCA  requirements. 

DoD  CIO  COMMENTS:  Concur. 

The  appropriate  DoD  officials  did  consider  the  limitations  directly  affecting  the  areas  of 
business  process  reengineering,  analysis  of  alternatives,  and  performance  measures  during  the 
CCA  certification  review  for  Increment  3  (Milestone  IIIc).  These  officials  reached  a  similar 
position  as  the  DoDIG.  They  also  concluded  that  because  RCAS  was  in  an  advanced  stage  of 
development  and  deployment  when  CCA  was  enacted,  it  most  likely  would  not  fully  meet  the 
tenets  of  the  CCA  since  the  opportunities  to  realize  the  most  substantial  benefits  from  “up 
front”  efforts  such  as  business  process  reengineering  or  analysis  of  alternatives  were  reduced 
by  that  time.  Nevertheless,  the  RCAS  PMO  and  the  functional  proponents  did  attempt  to 
redress  deficiencies  in  administrative  processes  that  were  adversely  affecting  the  performance 
of  a  core  mission  -  namely  mobilization.  In  addition,  the  RCAS  certification  package  was 
provided  in  draft  to  congressional  staff  for  review  and  comment.  Their  comments  were 
incorporated  into  the  certification  process  and  briefed  to  senior  staff. 

During  the  CCA  certification  review  for  Increment  4/5  (Milestone  Hid),  except  for 
determining  that  the  sustainment  requirements  for  RCAS  lacked  sufficient  funding,  the  OSD 
staff  determined  that  no  major  changes  or  issues  occurred  since  the  initial  certification.  We 
identified  affordability  explicitly  in  the  updated  certification  report  as  a  major  issue  that  had  a 
direct  bearing  on  whether  the  performance  goals  could  be  achieved.  While  the  affordability 
was  not  an  issue  for  the  earlier  certification,  we  determined  that  a  subsequent  POM  cycle  did 


29 


not  fund  needed  sustainment  requirements.  In  an  acquisition  decision  memorandum  of  July  2, 
2001,  the  DoD  CIO  directed  Army  to  conduct  an  affordability  analysis  before  the  next  fielding 
decision.  On  February  25,  2002,  the  Army  CIO  reported  that  the  analysis  has  been  conducted 
and  indicates  that  the  RCAS  program  is  affordable. 

•  RECOMMENDATION  1.  We  recommend  that  the  Chief  Information  Officer,  DoD,  to  best 
meet  the  intent  of  Clinger-Cohen  Act,  establish  policy  to  fully  report  program  compliance 
limitations. 

DoD  CIO  Comments:  Concur. 

We  will  continue  to  work  with  the  Component  CIOs  to  develop  specific  criteria  to  enable 
uniform  assessments  of  CCA  compliance.  In  2001,  we  established  a  DoD  CCA  Certification 
Report  Integrated  Process  Team  (IPT)  that  developed  a  CCA  certification  and  confirmation 
template  that  updated  the  information  contained  in  our  memorandum  of  July  13,  2000, 
“Department  of  Defense  (DoD)  Information  Technology  (IT)  Systems  Certification 
Requirements  ”  These  efforts  resulted  in  two  web  sites  being  established  to  further  enhance  the 
procedures  and  approach  to  be  used  by  DoD  Components  for  determining  whether  major 
automated  information  systems  are  developed  in  accordance  with  the  Clinger-Cohen  Act  of 
1996. 

If  you  have  any  question  on  the  above,  please  direct  them  to  my  action  officer,  Mr.  Edward 
Wingfield,  at  (703)  602-0980  x  126  or  ed.wingfield@osd.mil. 


(or  Priscilla  E.  Guthrie  * 

Deputy  Chief  Information  Officer 


cc:  Chief,  National  Guard  Bureau 

Chief  Information  Officer,  Department  of  the  Army 

Assistant  Secretary  of  the  Army  for  Acquisition,  Logistics,  and  Technology 


30 


National  Guard  Bureau  Comments 


DEPARTMENTS  OF  THE  ARMY  AND  THE  AIR  FORCE 

NATIONAL  GUARD  BUREAU 
1411  JEFFERSON  DAVIS  HIGHWAY 
ARLINGTON,  VA  22202-3231 


NGB-CIO 


22  March  2002 


MEMORANDUM  FOR  MARY  L.  UGONE,  DEPUTY  DIRECTOR  ACQUISITION 
MANAGEMENT  DIRECTORATE,  INSPECTOR  GENERAL,  DEPARTMENT  OF 
DEFENSE,  400  ARMY  NAVY  DRIVE,  ARLINGTON,  VA  22202 

SUBJECT:  Audit  Report  on  Certification  of  the  Reserve  Component  Automation 
System  (Project  No.  D2000AS-021 2.001) 


1 .  Reference  memorandum,  DOD  IG,  23  January  2002,  subject  as  above. 

2.  Enclosure  1  is  the  response  to  the  tentative  findings  and  recommendations 
contained  in  the  draft  report. 

3.  The  point  of  contact  in  my  office  is  Judith  Peterson,  DSN  327-3369  or  commercial 
(703)  607-3369,  email:  judy.peterson@pmorcas.ngb.army.mil 

FOR  THE  CHIEF,  NATIONAL  GUARD  BUREAU 

Enel  MAUREEN  T.  LISCHKE 

Chief  Information  Officer  and 
Program  Executive  Officer  for 
Information  Systems 

CF: 

CNGB 

DARNG 
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Response  to  the  Office  of  the  Inspector  General,  DoD  Draft  Audit  Report 
Clinger-Cohen  Act  Certification  of  the  Reserve  Component  Automation  System  (RCAS) 
Project  No.  D2000AS-02 12.001,  January  23,  2002 


NGB  CIO/PEO  IS  comments  are  provided  below  under  three  major  categories: 

I  Comments  on  Audit  Report  Findings; 

II  Comments  on  Audit  Report  Recommendations;  and 

III  General  Comments  on  Facts  as  Presented  in  the  Draft  Audit  Report. 

I.  Comments  on  Audit  Report  Findings 
I-A.  DOD  IG  Finding  -  Introduction  (Page  4): 

Limitations  of  the  RCAS  Program  {Project}  efforts  for  Increments  3  and  4/5  for 
compliance  with  the  intent  of  the  CCA  were  not  reported  by  the  CIO,  DoD.  The 
limitations  directly  affected  three  of  the  five  interest  items  that  were  specified  in  Section 
8121(b)(2).  This  condition  occurred  because  the  CIO  did  not  believe  that  the  weaknesses 
associated  with  business  process  reengineering  and  the  analysis  of  alternatives  were 
significant  enough  to  withhold  the  CCA  certification.  In  addition,  the  CIO  did  not 
identify  any  weakness  in  performance  measures  even  though  the  RCAS  PMO  and  the 
Reserve  Components  had  not  established  functional  performance  measures. 

NGB  CIO/PEO  IS  Comments:  Non-concur 

Required  activities  were  completed  before  the  CCA  was  enacted,  following  the 
regulations,  guidance,  and  best  practices  that  were  available  at  the  time,  many  of  which 
became  incorporated  into  the  CCA.  The  Project  has  provided  extensive  documentation  in 
this  regard.  Further,  the  following  comments  will  demonstrate  that  RCAS  properly 
addressed  all  five  interest  items  including  Business  Process  Reengineering,  Analysis  of 
Alternatives,  and  Performance  Measures. 

I-B.  DOD  IG  Finding  on  RCAS  Certification  Process  (Page  4): 

During  its  briefing  to  the  CIO,  the  review  team  presented  a  qualified  statement  for 
actions  regarding  business  process  reengineering  and  the  analysis  of  alternatives 
because  business  process  workflows  had  not  been  substantially  redesigned.  Additionally, 
the  PMO  had  not  considered  a  full  range  of  approaches  for  the  analysis  of  alternatives  to 
reduce  costs,  outsource  services,  streamline  operations,  or  privatize  functions. 

NGB  CIO/PEO  IS  Comments:  Non-concur 

The  Project  has  provided  an  extensive  set  of  artifacts  in  regard  to  the  occurrence  of  and 
content  of  these  activities. 

I-C.  DOD  IG  Finding  on  Business  Process  Reengineering  (Page  6): 
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Although  representatives  from  the  offices  of  the  ASD  (C3I)  and  the  Director,  PA&E, 
acknowledged  that  DoD  and  the  Army  had  not  focused  on  the  use  of  activity-based 
costing  techniques  to  simplify  or  otherwise  redesign  business  processes  before  investing 
in  RCAS,  the  certification  report  to  Congress  did  not  clearly  explain  that  RCAS  business 
process  reengineering  did  not  completely  meet  the  full  intent  of  the  CCA. 

NGB  CIO/PEO  IS  Comments:  Non-concur 

While  we  agree  that  activity-based  costing  (ABC)  techniques  provide  additional  insight 
into  an  agency’s  processes,  we  don’t  believe  that  the  absence  of  ABC  is  an  indicator  that 
Business  Process  Reengineering  (BPR)  did  not  occur.  The  Clinger-Cohen  Act  requires 
agencies  to  revise  (doesn’t  use  the  term  BPR)  mission-related  processes  and  does  not 
mandate  specific  techniques  to  utilize.  The  Project  provided  a  variety  of  artifacts  that 
documented  that  the  Army  Reserve  Component  refined  its  business  processes  prior  to  any 
significant  RCAS  investment. 


I-D.  DOD  IG  Finding  on  Business  Process  Reengineering  (Page  6): 

Although  the  RCAS  investment  should  improve  and  support  work  processes  to  reduce 
cost ,  improve  effectiveness,  and  implement  Government  and  commercial  off-the-shelf 
technology,  the  work  processes  of  the  Reserve  Components  were  not  fully  subjected  to 
business  process  reengineering .... 

Efforts  undertaken  by  the  RCAS  PMO  to  justify  information  technology  investments  in  the 
system  did  not  identify,  dramatically  redesign,  and  eliminate  low  or  no  value-added 
functions  or  work  processes  before  deciding  to  invest  in  RCAS.  ...  The  efforts  of  RCAS 
stakeholders  and  program  officials  related  to  business  process  reengineering  could,  at 
best,  be  considered  an  improvement  in  the  functional  process;  however,  they  could  not  be 
considered  a  redesign  and  reengineering  of  established  business  processes  or  workflows. 

NGB  CIO/PEO  IS  Comments:  Non-concur 

RCAS  conducted  significant  business  process  reengineering  efforts  during  the  early 
stages  of  the  Project.  Beginning  in  1989,  the  PMO  facilitated  user  workshops  with  over 
500  Army  National  Guard  (ARNG),  U.S.  Army  Reserve  (USAR),  and  Active  Army 
functional  users  to  ensure  all  business  processes  were  accurately  created  (many  didn’t 
exist),  refined,  and  documented.  Additionally,  no  value/low  value  processes  were 
eliminated  during  these  workshops.  The  RCAS  Functional  Description  (FD)(July  1995), 
documents  the  results  of  these  activities  in  25  volumes.  During  the  Project’s  Restructure, 
the  functional  requirements  of  the  FD  were  consolidated  into  the  Operational  Concept 
Document  (OCD).  The  OCD  was  utilized  to  create  the  RCAS  Enterprise  Model,  which 
documents  the  RC’s  information  needs  and  business  processes  across  1 1  functional  areas. 
From  the  Enterprise  Model,  Use  Cases  (11  Volumes  documenting  pre-automation 
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business  processes)  were  developed  for  each  functional  area,  which  ultimately  formed  the 
foundation  of  the  RCAS  design  requirements.  Thus,  before  RCAS  design  requirements 
were  translated  into  an  Information  Technology  (IT)  investment,  the  RC  functional 
communities  had  created,  refined,  and  documented  core  RC  business  processes.  These 
efforts  fundamentally  changed  the  way  the  ARNG  and  the  USAR  conducted  business  in 
the  functional  areas  that  RCAS  addresses  and  meet  the  intent  of  CCA. 

I-E.  DOD  IG  Finding  on  Business  Process  Reengineering  (Page  7): 

In  essence,  the  PMO,  in  conjunction  with  the  functional  proponents,  had  not  performed 
business  process  reengineering  on  any  of  the  RCAS  increments  since  the  1996  enactment 
of  the  CCA. 

NGB  CIO/PEO  IS  Comments:  Conditionally  Concur 

Although  we  agree  that  the  statement  is  true,  by  1 996  RCAS  had  already  been 
restructured,  the  requirements  for  all  increments  were  set,  and  Milestone  II  approval  had 
been  received.  The  Milestone  III  increment  decisions  are  not  an  appropriate  time  to 
conduct  BPR  since  they  are  merely  the  extension  of  the  earlier  decisions.  Therefore,  it 
should  not  be  expected  that  RCAS  or  the  functional  users  would  conduct  additional  BPR 
on  any  of  the  increments,  regardless  of  their  occurring  after  the  enactment  of  the  CCA. 
The  DOD  IG  in  its  conclusion  to  the  audit  (page  14)  states  that  RCAS  was  past  the  stage 
where  business  process  reengineering  would  be  useful. 

I-F.  DOD  IG  Finding  on  Business  Process  Reengineering  (Page  7): 

The  PMO  exerted  extensive  efforts  to  overcome  the  inadequacies  of  existing  methods  and 
procedures  by  proposing  to  automate  inefficient,  functionally  disconnected,  and  manual 
processes. 

NGB  CIO/PEO  IS  Comments:  Non-concur 

The  Project  provided  substantial  evidence  that  the  Project,  in  concert  with  the  user 
community,  analyzed  alternatives  and  refined  business  processes  prior  to  crafting  the 
technical  solution. 

I-G.  DOD  IG  Finding  on  Analysis  of  Alternatives  (Page  7): 

The  PA&E  office  qualified  its  assessment  of  the  analysis  of  alternatives  in  the  February 
2000  briefing  to  the  CIO  because  the  August  1996  analysis  of  alternatives  did  not 
consider  a  full  range  of  alternatives  to  reduce  cost,  such  as  outsourcing  specific 
functions,  streamlining  or  privatizing  routine  administrative  processes. 

NGB  CIO/PEO  IS  Comments:  Non-concur 
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The  Project’s  restructure  in  1995  was  the  result  of  a  major  Analysis  of  Alternatives 
(AoA)  documented  in  the  Validation  Assessment  Team’s  (VAT)  Final  Report.  It 
demonstrates  that,  beyond  exploring  several  alternatives,  the  VAT  evaluated  multiple 
technical  solutions,  Project  organizations,  and  day-to-day  business  processes  before 
selecting  the  RCAS  solution. 

I-H.  DOD  IG  Finding  on  Analysis  of  Alternatives  (Page  8): 

The  PMO  stated  that  RCAS  was  exempt  from  outsourcing  routine  administrative 
processes  because  the  system  was  established  under  title  10,  United  States  Code,  Armed 
Services  Program  because  it  supported  numerous  inherent  Government  functions,  such 
as  manning,  equipping,  training,  and  sustaining  the  Army’s  Reserve  Components.  ...  The 
PMO  did  not  provide  any  documentation  to  show  how  RCAS  was  an  inherent  government 
function. 

NGB  CIO/PEO  IS  Comments:  Non-concur 

The  PMO  stated  on  April  3,  2001,  “RCAS  supports  the  Title  10  functions  of  manning, 
equipping,  training,  and  sustaining  the  Army  Reserve  Component.”  In  accordance  with 
the  Federal  Activities  Inventory  Reform  (FAIR)  Act,  agencies  annually  identify  those 
activities  not  considered  inherently  governmental.  To  date,  the  functions  and  jobs  that 
RCAS  supports  are  not  known  to  be  identified  on  the  Army's  FAIR  Act  inventories. 

Also,  the  National  Guard  outsources  to  the  states  those  functions  not  uniquely  military  or 
inherently  governmental.  In  the  case  of  both  the  National  Guard  and  Reserve,  military 
and  dual  status  technicians  staff  many  of  the  remaining  functions.  Technicians  are 
required  by  law  to  be  members  of  the  Army  Reserve  Component  as  well. 

I-I.  POD  IG  Finding  on  Economic  Analysis  (Page  8): 

However,  we  noted  that  the  benefits  used  in  the  computation  of  the  return  on  investment 
consisted  of  “ soft  dollars  ”  or  benefits  that  could  not  be  quantitatively  tracked.  The 
PA&E  office  questioned  the  amount  of  actual  benefits  because  it  represented 
unquantifiable  benefits .  Because  the  return  on  investment  was  based  on  unquantifiable 
benefits,  the  use  of performance  measures  to  assess  the  benefits  of  the  RCAS  investment 
becomes  even  more  important. 

NGB  CIO/PEO  IS  Comments:  Non-concur 

RCAS  has  quantified  the  productivity  improvements  derived  by  the  project.  The  Project 
has  accomplished  seven  cost  benefit  analyses  (CBA)  in  the  last  six  years.  To  determine 
the  Project’s  Return  on  Investment  (ROI),  RCAS  cost-savings  and  productivity 
improvements  have  been  quantitatively  expressed  by  evaluating  labor  requirements,  cycle 
time,  frequency,  and  output  quality  for  RCAS  business  processes.  Based  on  the  Project’s 
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CBA  strategy,  the  PMO  also  performs  post-implementation  reviews  to  quantitatively 
track  the  actual  realization  of  system  benefits  (i.e.,  cost-savings  and  productivity 
improvements).  All  of  the  RCAS  Economic  Analyses  quantify  the  benefits  of  RCAS  and 
reveal  a  substantial  ROI.  RCAS  also  generates  non-quantifiable  benefits  such  as 
improved  timeliness  and  accuracy  of  data.  These  are  identified  in  the  CBA  but  not  used 
to  calculate  the  ROI. 

I-J  .  DOD  IG  Finding  on  Performance  Measures  (Page  8): 

Functional  Proponents  of  RCAS  did  not  establish  a  performance  measurement  plan  to 
assess  performance  or  to  identify  whether  the  desired  results  were  being  achieved . 
Specifically,  new  processes  were  not  compared  against  measures  of  outcome,  output,  and 
efficiency  of  RCAS. 

NGB  CIO/PEO  IS  Comments:  Non-concur 

The  National  Guard  Bureau  and  Army  Reserve  functional  proponents  captured  key 
functional  requirements  in  the  RCAS  Operational  Concept  Document  (OCD)  during  the 
Project’s  restructure  in  1995.  These  included  six  major  RCAS  functional/technical 
performance  measure  categories:  1)  Administrative  Operations,  2)  Interoperability,  3) 
Logistics,  4)  Security,  5)  MANPRINT,  and  6)  Reliability.  While  there  were  no  metrics 
established  to  quantify  functional  performance  at  that  time,  it  was  uniformly  understood 
that  accomplishment  of  the  OCD  requirements  would  dramatically  improve  business 
processes. 

The  RCAS  Project  Manager  (PM)  then  incorporated  these  requirements  into  the  RCAS 
Acquisition  Program  Baseline  (APB).  The  PM  worked  with  the  user  community  to 
develop  key  performance  parameters  (KPP)  and  measures  of  effectiveness  (MOE),  which 
included  many  functional  performance  metrics.  Examples  include  mobilization  order 
processing  times,  logistics  response  times,  and  maintenance  response  times. 

To  independently  measure  the  Project’s  ability  to  meet  these  functional/technical 
performance  measures,  the  National  Guard  Bureau  partners  with  the  Army  Test  and 
Evaluation  Command  (ATEC)  to  measure  functional/technical  performance  via  both 
operational  and  mobilization  activities.  In  addition,  the  National  Guard  Bureau  directs 
independent  benefit  analyses  to  be  accomplished  on  each  increment  of  RCAS 
functionality.  These  benefit  analyses  periodically  measure  the  actual  functional 
processes  to  identify  process  improvements,  cycle  times,  cost-savings,  and  labor 
requirements.  To  date,  the  National  Guard  Bureau  has  measured  100+  functional 
processes.  Examples  include:  retirement  eligibility,  organizational  authority  publication, 
and  medical  readiness. 

This  approach  provides  the  PM  and  the  user  community  with  both  functional  and  system 
performance  measures,  which  is  consistent  with  the  intent  of  the  CCA. 
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I-K.  DOD  IG  Finding  on  Benchmarks  (Page  9): 

Although  the  CIO  Certification  report  stated  that  benchmarks  were  used  to  derive  risk- 
adjusted  alternative  technical  solutions  during  the  1995  restructure  of  the  program,  the 
PMO  did  not  provide  any  documentation  to  support  its  benchmarking  efforts. 

NGB  CIO/PEO  IS  Comments:  Non-concur 

The  Reserve  Component  (RC)  functional  communities  used  benchmarks  during  the  user 
workshops  conducted  from  1985  through  1989.  They  benchmarked  against  each  other 
(e.g.,  Iowa  compared  to  Indiana,  National  Guard  process  compared  to  Army  Reserve 
process)  and  in  concert  with  federal  agencies  that  might  be  involved  in  the  same  or 
similar  business  processes  (e.g.,  OSHA,  VA).  These  workshops  resulted  in  the  refined 
processes  documented  in  the  RCAS  Functional  Description  (FD),  which  form  the 
foundation  of  RCAS  requirements.  A  complete  trace  of  this  process  was  provided  to 
DOD  IG  earlier  in  the  audit. 

In  addition,  during  the  Project’s  restructure,  benchmarking,  modeling,  trade-off  analyses, 
and  user  expectation  management  techniques  were  utilized  to  derive  low,  medium,  and 
high-risk  alternatives.  As  one  example,  the  RCAS  Validation  Assessment  Team  (VAT) 
conducted  benchmark  studies  during  the  analysis  of  alternative  sustainment  strategies. 
The  VAT,  using  a  “Benchmark  Interview  Guide,”  evaluated  and  sought  suggestions  from 
four  existing  Government  programs  and  at  least  fifteen  commercial  vendors.  This 
documentation  was  provided  separately  to  the  DOD  IG  on  21  March  2002. 

I-L.  DOD  Finding  on  Integrating  Integrated  Product  Team  (Page  12) 

As  of  January  2002,  the  review  (Milestone  IHe)  had  not  occurred. 

NGB  CIO/PEO  IS  Comments:  Non-concur 

The  Milestone  Ille  review  occurred  in  October  2001 .  As  of  January  2002,  the 
Acquisition  Decision  Memorandum  had  not  been  issued. 

I-M.  DOD  IG  Finding  on  Milestone  Exit  Criteria  (Page  12): 

Although  the  PMO  and  the  Army  Reserve  provided  an  updated  training  strategy  on 
March  8,  1999  {1998},  as  directed  in  the  January  8,  1998,  Acquisition  Decision 
Memorandum  for  Milestone  Illb  (Increment  2),  the  strategy  did  not  include  a  schedule 
for  implementing  training  improvements... 
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In  its  March  2000  assessment  of  Increment  3,  the  Army  Test  and  Evaluation  Command 
identified  training  as  a  major  problem  area  and  stated  that  user  training  must  be 
significantly  improved .... 

Although  the  PMO  and  the  Army  Reserve  provided  a  revised  training  strategy ,  they  still 
need  to  develop  a  viable  training  plan,  for  Army  approval,  that  provides  the  required 
training  and  documentation  support  for  RCAS  users,  system  administrators,  and 
command-level  representatives  and  a  schedule  for  implementing  the  training. 

NGB  CIO/PEO  IS  Comments:  Non-concur 

The  Office  of  the  Chief,  Army  Reserve  developed  a  training  strategy  within  60  days 
leading  to  the  improvement  and  expansion  of  COTS  training  as  directed  by  the 
Acquisition  Decision  Memorandum  (ADM)  for  Increment  2.  The  strategy  and  schedule 
are  contained  with  the  “Reserve  Component  Automation  System  Training  Plan” 
published  by  the  Army  Reserve  Chief  Information  Office  (BG  Helmly)  on  March  8, 

1998.  Paragraph  5  of  the  plan  defines  the  schedule  for  implementing  improved  training 
in  three  phases.  This  sequential  schedule  provides  the  necessary  guidance  for  unit 
commanders  to  develop  their  individual  soldiers’  training  plans.  This  fully  met  the 
requirements  and  intent  of  the  ADM  at  the  Milestone  IIIc  review.  This  document  was 
provided  to  the  DOD  IG. 

The  Project,  with  the  full  support  of  the  users,  has  developed  a  “viable  training  plan”  for 
each  Increment.  The  plan  and  its  implementation  have  matured  with  the  Project.  The 
training  strategy  for  Increment  2  (primarily  COTS)  was  based  primarily  on  unit 
evaluation  and  individual  self-paced  instruction  and  resulted  in  the  ADM  direction. 
Increment  3  began  the  fielding  of  RCAS-developed  applications  and  with  it  a  revision  to 
the  training  strategy.  As  a  result  of  the  Army  Test  and  Evaluation  Command’s  (ATEC) 
assessment  of  Increment  3  training,  the  Project  determined  that  self-instruction  on  RCAS- 
developed  applications,  with  users  relying  primarily  on  a  Software  User’s 
Manual/Training  Materials  (SUM/T)  (project  developed  training  documentation)  was 
insufficient.  Therefore,  with  Increment  4/5,  Programs  of  Instruction  (POI)  were 
developed  for  RCAS-developed  applications  and  the  Classified  Capable  Workstation 
(CCW),  and  OIS  site  users  received  hands-on  classroom  instruction.  Additionally, 
Database  and  Installation  Workshops  were  instituted  and  a  modularized  RCAS 
Administrator  POI  was  developed.  The  training  enhancements  instituted  with  Increment 
4/5  continued  into  Increment  6  and  beyond.  In  addition,  the  Project’s  most  recent 
training  enhancement,  the  Train-the-Trainer  Program,  will  further  improve  the  training 
process  and  increase  user  skill  levels  during  full  fielding  of  RCAS  developed- 
applications.  Draft  schedules  are  attached  and  will  be  incorporated  into  the  next  RCAS 
Training  and  Documentation  Plan. 

The  RCAS  Training  Plan  is  approved  by  the  Reserve  Component  RCAS  Training 
Working-Level  Integrated  Process  Team  (WIPT).  Members  of  the  WIPT  include 
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personnel  representing  the  Army  National  Guard  ( ARNG)  and  the  United  States  Army 
Reserve  (USAR)  Chief  Information  Officers,  the  ARNG  and  the  USAR  Customer  Focus 
Teams,  the  ARNG  and  USAR  functional  proponents,  and  RCAS  PMO  Training  and 
Documentation  staff  members. 

The  Project  does  provide  the  required  training  and  documentation  support  for  RCAS 
users,  system  administrators,  and  command-level  representatives  which  the  Army  Test 
and  Evaluation  Command  (ATEC)  evaluates  during  each  increment’s  operational  test 
(OT).  For  Increment  7,  the  Project  will  field  forty-nine  (49)  documents. 

As  a  result  of  ATEC’s  evaluation  of  Increment  3  training  and  lessons  learned  at  the  OIS 
units,  training  improvements  were  implemented  as  described  above.  During  the  recent 
evaluation  of  RCAS  Increment  6,  ATEC  rated  “Training”  as  one  of  the  Project’s  areas  of 
strength.  At  the  7  September  2001  RCAS  Increment  6  Limited  User  Test  Emerging 
Results  Briefing,  ATEC  noted  in  its  key  findings  that  98%  of  functional  users  rated 
training  as  “Very  Effective”  or  “Effective”;  97%  of  functional  users  rated  the  SUM/T  as 
adequate;  and  86%  of  the  System  Administrators  and  Database  Administrators  rated 
training  as  “Effective”.  All  of  the  functional  users  said  they  have  the  necessary  basic 
Windows  skills  to  use  the  applications. 

Clearly,  the  improved  training  first  documented  in  March  1 998  has  been  successfully 
implemented  and  it  has  been  further  improved.  Many  schedules  have  been  executed. 

The  intent  of  the  ADM  was  met  by  the  USAR  in  March  1998. 

I-N.  DOD  IG  Finding  on  Conclusion  (Page  14): 

Business  process  reengineering  and  an  analysis  of  alternatives  would  still  be  useful  for 
program  increments  that  include  new  functions  and  should  be  applied  at  the  start  of  each 
new  increment  to  ensure  that  all  new  functions  comply  with  the  CCA . 

NGB  CIO/PEO  IS  Comments:  Non-Concur 

Consistent  with  DoD  5000  guidance,  “new”  functions  have  not  entered  the  RCAS 
production  process  after  its  Milestone  (MS)  C  decision.  RCAS’  MS  C  (formerly  MS  III) 
decision  was  in  1996  and  authorized  the  Project  to  commence  full-rate  production. 
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II.  Comments  on  Audit  Report  Recommendations 
Recommendation  2: 

We  recommend  that  the  Chief,  National  Guard  Bureau,  and  the  Chief,  Army  Reserve, 
review  the  five  Section  8121(b)(2)  interest  items  for  the  proposed  capabilities  of 
Increment  6,  as  appropriate,  and  for  any  future  increments,  and  determine  whether  the 
selected  solution  complies  with  the  intent  of  the  requirements  of  the  Clinger-Cohen  Act. 

NGB  CIO/PEO  IS  Comments:  Conditionally  Concur 

The  Chief,  National  Guard  Bureau  has  reviewed  the  previous  two  RCAS  Clinger-Cohen 
Act  compliance  submissions  and  will  continue  to  scrutinize  the  Project’s  future 
submissions  through  the  end  of  the  acquisition  program. 

BPR  and  AoA  were  conducted  during  the  earlier  phases  of  system  development.  The 
incremental  reviews  performed  as  a  part  of  the  RCAS  Increments  (releases)  are  a 
validation  of  the  earlier  Milestone  decisions  by  ensuring  that  the  requirements  are 
satisfied  with  the  parameters  of  the  Project’s  baseline.  RCAS  General  Officers'  Steering 
Committee  (GOSC)  have  reviewed  and  approved  each  increment  throughout  the 
development  and  fielding  of  the  system.  The  Chief,  National  Guard  Bureau  chairs  the 
GOSC  with  participation  by  the  Director,  Army  National  Guard,  Chief  Army  Reserve, 
the  DISC4  (G6)  and  Department  of  the  Army,  DCSOPS. 

Recommendation  3a: 

We  recommend  that  the  Chief  National  Guard  Bureau,  and  the  Chief,  Army  Reserve 
require  functional  proponents  of  the  Reserve  Component  Automation  System  to  establish 
functional  performance  measures  to  assess  performance  and  achievement  of  desired 
results,  and  continually  compare  new  processes  of  outcome,  output,  and  efficiency  after 
the  deployment  of  the  Reserve  Component  Automation  System. 

NGB  CIO/PEO  IS  Comments:  Concur  in  principle 

We  agree  performance  measures  are  important  to  establish  and,  in  fact,  performance 
measures  (system  and  functional)  established  by  the  functional  proponents  do  exist  and 
are  documented  in  several  Project  documents  (e.g.,  Acquisition  Program  Baseline  (APB), 
section  3.1.5).  Beyond  technical  performance  parameters,  RCAS  has  administrative, 
interoperability,  logistics,  and  security  performance  measures.  They  quantitatively  set 
standards  for  hundreds  of  RC  processes  and  compare  the  labor  requirements,  process 
flow,  task  duration,  and  frequency  of  the  new  RCAS  business  processes  to  the  pre-RCAS 
business  process.  Additionally,  the  Army  Reserve  CIO  continues  to  measure  their  IT 
enterprise,  which  includes  RCAS  as  part  of  their  portfolio.  The  current  approach 
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provides  both  functional  and  system  performance  measures  and  an  assessment  of 
outcome,  consistent  with  the  Clinger-Cohen  Act. 

Recommendation  3b: 

We  recommend  that  the  Chief,  National  Guard  Bureau,  and  the  Chief,  Army  Reserve 
reassess  the  risk  of  exchanging  unencrypted  files  containing  sensitive  data  between  the 
Reserve  Component  Automation  System  for  proposed  and  fielded  increments  and  other 
networked  systems  and,  if  appropriate,  implement  encryption  technology. 

NGB  CIO/PEO  IS  Comments:  Concur  in  principle. 

As  the  Designating  Approving  Authority  (DAA)  for  RCAS,  the  Director  of  Information 
Systems  for  Command,  Control,  Communications,  and  Computers  (DISC4),  in 
coordination  with  representatives  from  the  Chief,  National  Guard  Bureau  and  the 
Program  Management  Office,  assessed  the  risks  of  not  encrypting  RCAS  file 
transmissions  during  the  1999  RCAS  Defense  Information  Technology  Security 
Certification  and  Accreditation  Process  (DITSCAP)  and  deemed  the  risk  of  potential 
harm  low.  The  Designated  Approval  Authority  will  reassess  these  risks  in  the  course  of 
the  next  DITSCAP  accreditation,  currently  scheduled  for  November  2002.  A  separate 
action,  outside  the  directed  process,  by  the  Chief,  National  Guard  Bureau  and  the  Chief, 
Army  Reserve  is  not  required.  However,  we  continue  to  keep  security  foremost  and  are 
pursuing  other  technologies  such  as  utilizing  Virtual  Private  Network  and  incorporating 
Private  Key  Infrastructure  and  Common  Access  Card  into  the  infrastructure. 

Recommendation  4a: 

We  recommend  that  the  Project  Management  Officer,  Reserve  Component  Automation 
System,  develop  a  plan  for  the  approval  of  the  Chief  Information  Officer,  Department  of 
the  Army,  that  provides  required  training  and  documentation  support  for  Reserve 
Component  Automation  System  users,  system  administrators,  and  command  level 
representatives. 

NGB  CIO/PEO  IS  Comments:  Non-concur. 

Training  has  and  will  continue  to  be  accomplished  in  accordance  with  existing 
acquisition  procedures  and  Army  doctrine,  with  the  involvement  of  personnel  from  the 
Army  Reserve  Component  CIOs.  The  Project  has  established  a  viable  training  plan  for 
each  RCAS  increment,  which  has  been  approved  by  the  RCAS  Training  Working 
Integrated  Product  Team  (WIPT)  and  provides  the  required  training  and  documentation 
for  each  level  of  RCAS  users.  The  Army  Test  and  Evaluation  Command  (ATEC) 
evaluates  this  documentation  during  each  increment’s  Operational  Test  (OT).  During  the 
recent  evaluation  of  RCAS  Increment  6,  ATEC  rated  “Training”  as  one  of  the  Project’s 
areas  of  strength.  In  addition,  the  Project’s  most  recent  training  enhancement,  the  Train- 
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the-Trainer  Program,  will  further  improve  the  training  process  and  increase  user  skill 
levels.  The  Project  has  successfully  demonstrated  the  strength  of  its  training  program. 
Additional  artifacts  were  provided  to  the  DODIG  on  21  March  2002. 

Recommendation  4b: 

We  recommend  that  the  Project  Management  Officer,  Reserve  Component  Automation 
System,  develop  a  plan  for  the  approval  of  the  Chief  Information  Officer,  Department  of 
the  Army,  that  funds  the  operation  and  support  of  the  Reserve  Component  Automation 
System  for  FYs  2002  through  2007. 

NGB  CIO/PE O  IS  Comments:  Conditionally  Concur. 

This  is  guidance  provided  by  OSD  in  the  Milestone  (MS)  Hid  Acquisition  Decision 
Memorandum  (ADM).  The  RCAS  PM  and  Army  CIO  have  worked  jointly  with  the 
ARNG  and  USAR  to  identify  critical  system  sustainment  requirements,  assess 
sustainment  risks,  develop  a  revised  Program  Office  Estimate  (POE),  and  gamer  required 
funding.  In  recent  program  budget  events  and  with  the  participation  of  the  office  of  the 
Army  CIO,  the  Army  directed  over  $300M  to  address  Project  shortfalls.  As  a  result  of 
this  and  the  Project’s  affordability  analysis,  the  Office  of  the  Deputy  Chief  of  Staff  for 
Program  Analysis  and  Evaluation  declared  the  Project  as  affordable. 

III.  General  Comments  on  Facts  as  Presented  in  the  Audit  Report 

The  following  comments  are  presented  to  improve  the  document’s  accuracy: 

Page  i,  Introduction  -  RCAS  continues  to  have  7  increments  and  7  milestones. 

Page  ii.  Summary  of  Recommendations  -  The  RCAS  PMO  is  the  Project  Management 
Office  and  is  led  by  a  Project  Manager  -  there  isn’t  a  Project  Management  Officer. 

Page  ii,  Summary  of  Recommendations  -  the  Reserve  Components  should  be  referred  to 
here  as  the  Army  Reserve  Component,  which  consists  of  the  Army  National  Guard  and 
the  Army  Reserve. 

Page  2,  RCAS  is  a  Project,  not  a  Program. 

Page  2-3,  Change  to  read  “Beyond  FY  2002, ...  users  and  RCAS  software  maintainer. 

Page  12,  The  updated  training  strategy  was  provided  on  March  8,  1998  vice  March  8, 
1999. 

Page  16,  Scope  and  Methodology  -  Compliance  report  was  prepared  by  the  RCAS  PMO 
not  by  the  Army  National  Guard. 
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Page  20,  Department  of  the  Army  -  The  Program  Executive  Officer,  Reserve  Component 
Automation  System  should  be  referred  to  as  the  Program  Executive  Office  for 
Information  Systems. 
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DRAFT  2002  TRAIN-TIHE-TRAINER  SCHEDULE 


DATE 

APPLICATION 

LOCATION 

19-21  March 

SOH 

NG-PEC 

26  March 

RPAM 

NG  -  PEC 

2-4  April 

SOH 

BBTS 

9  April 

UPS/CMS 

BBTS 

10  April 

MILPO/IDV-P 

BBTS 

16-18  April 

SOH 

NG  -  PEC 

16-18  April 

SOH 

BBTS 

23-25  April 

SOH 

NG  -  PEC 

23-25  April 

SOH 

BBTS 

7-9  May 

SOH 

ARRTC 

7-9  May 

SOH 

BBTS 

14-16  May 

SOH 

ARRTC 

14-16  May 

SOH 

BBTS 

21-23  May 

SOH 

NG-PEC 

21-23  May 

SOH 

BBTS 

29  May 

UPS/CMS 

BBTS 

30  May 

MILPO/IDV-P 

MILPO/IDV-P 

4-6  June 

SOH 

ARRTC 

4-6  June 

SOH 

BBTS 

11-13  June 

SOH 

NG-PEC 

11-13  June 

SOH 

ARRTC 

1 1  June 

UPS/CMS 

BBTS 

12  June 

MILPO/IDV-P 

BBTS 

25  June 

UPS/CMS 

BBTS 

26  June 

MILPO.IDV-P 

BBTS 

Dates  for  the  balance 

of  2002  classes  are  in  coordination. 

Atch 
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DEPARTMENT  OF  THE  ARMY 

OFFICE  OF  THE  CHIEF,  ARMY  RESERVE 
WASHINGTON,  DC  20310-2400 


25  Mar  02 


MEMORANDUM  FOR  Inspector  General,  Department  of  Defense,  400  Army  Navy  Drive, 
Arlington,  VA  22202-4704 

SUBJECT:  Department  of  Defense,  Inspector  General  (DODIG)  of  the  Process  Used  to  Certify 
that  Major  Automated  Information  Systems  are  Managed  in  Accordance  with  the  Clinger-Cohen 
Act,  Draft  Report  entitled  Certification  of  the  Reserve  Component  Automation  System ,  Project 
No.  D2000AS-0212.001,  23  Jan  02 


1 .  We  are  enclosing  our  Command  Comments  to  the  subject  draft  report. 

2.  If  you  have  any  questions  or  need  additional  information,  please  contact  Mr.  John  Price  at 
(404)  464-8191  or  email  him  at  pricejoh@usarc-emh2.army.mil. 

FOR  THE  CHIEF,  ARMY  RESERVE: 


Director,  Army  Reserve  Internal  Review 
and  Management  Control  Process 
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DODIG  Draft  Report 

Certification  of  the  Reserve  Component  Automation  System 
Army  Reserve  Command  Comments 


Finding.  Limitations  of  the  Reserve  Component  Automation  System  (RCAS)  Program  efforts 
for  Increments  3  and  4/5  for  compliance  with  the  intent  of  the  Clinger-Cohen  Act  (CCA)  were 
not  reported  by  the  CIO,  DOD.  The  limitations  directly  affected  three  of  the  five  interest  items 
that  were  specified  in  Section  8121(b)(2).  This  condition  occurred  because  the  CIO  did  not 
believe  that  the  weaknesses  associated  with  business  process  reengineering  and  the  analysis  of 
alternatives  were  significant  enough  to  withhold  the  CCA  certification.  In  addition,  the  CIO  did 
not  identify  any  weakness  in  performance  measures  even  though  the  RCAS  Project  Management 
Office  and  the  Reserve  Components  had  not  established  functional  performance  measures. 
Accordingly,  Congress  was  not  informed  that  the  RCAS  was  not  being  developed  in  full 
compliance  with  CCA  requirements. 

Recommendations  and  Command  Comments 

Recommendation  2.  We  recommend  that  the  Chief,  National  Guard  Bureau,  and  the  Chief, 
Army  Reserve,  review  the  five  Section  8121(b)(2)  interest  items  for  the  proposed  capabilities  of 
Increment  6,  as  appropriate,and  for  any  future  increments;  and  determine  whether  the  selected 
solution  complies  with  the  intent  of  the  requirements  of  the  Clinger-Cohen  Act. 

Army  Reserve  Comments.  Concur  with  the  intent  of  the  recommendation.  The  Army  Reserve 
is  dedicated  to  the  principles  of  Clinger-Cohen,  and  strives  to  apply  these  principles  in  all  aspects 
of  IT  management.  Moreover,  the  Army  Reserve  manages  its  network  and  associated  systems 
and  applications  as  an  enterprise.  Thus,  all  actions  taken  to  address  RCAS  shortfalls  will  be 
exercised  enterprise-wide.  In  other  words,  those  actions  taken  on  behalf  of  RCAS  will  be  the 
same  actions  taken  on  behalf  of  all  Army  Reserve  systems. 

Recommendation  3.  We  recommend  that  the  Chief,  National  Guard  Bureau,  and  the  Chief, 
Army  Reserve: 

a.  Require  functional  proponents  of  the  Reserve  Component  Automation  System  to  establish 
functional  performance  measures  to  assess  performance  and  achievement  of  desired  results,  and 
continually  compare  new  procedures  against  measures  of  outcome,  output,  and  efficiency  after 
the  deployment  of  Reserve  Component  Automation  System. 

Army  Reserve  Comments.  Concur  with  the  intent  of  the  recommendation.  The  Reserve 
Component  Automation  System  (RCAS)  supports  but  a  portion  of  the  business  processes  within 
each  of  the  functional  areas.  The  remainder  is  supported  by  a  myriad  of  Army  systems,  legacy 
applications,  and  individually  developed  products.  For  instance,  Force  Programs  functionality  is 
a  large  part  of  RCAS  Release  3.  Overall,  however,  the  Total  Army  Authorization  Document 
System  (TAADS),  the  Regional  Level  Application  Software  system  (RLAS),  and  others  support 
Force  Programs  business  processes,  in  addition  to  the  RCAS.  Measurement  of  RCAS  operations 
in  isolation  to  the  others  would  potentially  be  counterproductive  or  misleading. 
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Alternatively,  the  Army  Reserve  Chief  Information  Office  (CIO)  is  pursuing  a  more  holistic 
means  to  catalog,  and  measure  supporting  information  systems.  Begun  last  year,  the  CIO  is 
assembling  an  Information  Technology  (IT)  portfolio,  identifying  all  systems,  applications,  and 
supporting  IT  expense  areas  contributing  to  functional  mission  accomplishment.  Once  complete, 
this  portfolio  will  serve  as  a  baseline  for  ongoing  business  process  review  and  reengineering, 
application  and  system  consolidation,  and  performance  management.  As  a  contributor  to  several 
functional  areas,  RCAS  is  captured  in  the  Army  Reserve's  IT  portfolio. 

b.  Reassess  the  risk  of  exchanging  unencrypted  files  containing  sensitive  data  between  the 
Reserve  Component  Automated  System  for  proposed  and  fielded  increments  and  other 
networked  systems  and,  if  appropriate,  implement  encryption  technology. 

Army  Reserve  Comments.  Concur  with  the  intent  of  the  recommendation.  Data  exchanged 
between  the  RCAS  and  other  Army  Reserve  systems  occurs,  for  the  most  part,  within  the 
confines  of  the  Army  Reserve's  Intranet;  that  is,  inside  a  secure  network  boundary.  The 
likelihood  of  such  data  being  compromised,  or  intercepted,  is  minimized  by  its  residence  on  the 
Intranet.  On  a  larger  scale,  however,  the  scenario  whereby  RCAS,  or  any  other  Army  system, 
exchanges  data  with  another  system  or  application  "in  the  clear"  is  a  cause  for  concern. 

Recent  world  events,  coupled  with  Army  transformation  initiatives,  are  combining  to  radically 
alter  the  Army  Reserve  IT  landscape.  One  outcome  is  likely  to  be  a  consolidation  of 
applications,  systems,  and  key  IT  services  into  centralized  data  centers.  Access  into  and  out  of 
these  data  centers  will  be  tightly  controlled,  likely  utilizing  Virtual  Private  Network  (VPN) 
technology.  The  VPN  technologies,  coupled  with  strong  domain  authentication  in  place 
throughout  the  Army  Reserve,  eliminates  the  need  for  performance-robbing  encryption 
techniques. 

A  timeline  for  migration  of  all  applications  and  systems,  including  RCAS,  into  data  centers  has 
not  been  set.  The  migration  has  begun,  however  with  the  implementation  of  Active  Directory, 
Army  Knowledge  Online,  the  Global  Combat  Service  Support  -  Army,  the  Inspector  General 
Network,  and  others.  As  new  applications  are  developed,  or  older  ones  are  revised,  the  trend  is 
to  consolidate  servers,  centralize  operations,  and  thin  clients.  A  serendipitous  side  effect  of  this 
is  the  better  security  that  can  be  afforded  system-to-system  interfaces  in  a  consolidated 
environment. 


2 


Renumbered 

as 

Recommen¬ 
dation  4. 


47 


Army  CIO  Comments 


Final  Report 
Reference 


Renumbered 

as 

Recommen¬ 
dation  4. 


Renumbered 

as 

Recommen¬ 
dation  5. 


Office,  Direcfor  of  Information 
Systems  for  Command,  Control, 
Communications,  &  Computers 


DEPARTMENT  OF  THE  ARMY 
OFFICE  OF  THE  SECRETARY  OF  THE  ARMY 
107  ARMY  PENTAGON 
WASHINGTON  DC  20310-0107 


April  4,  2002 


MEMORANDUM  FOR  OFFICE  OF  THE  INSPECTOR  GENERAL,  DEPARTMENT  OF 
DEFENSE,  400  ARMY  NAVY  DRIVE  (ROOM  801),  ARLINGTON,  VA  22032-04707 

SUBJECT:  Audit  Report  on  Certification  of  the  Reserve  Component  Automation 
System  (RCAS)  (Project  No.  D2000AS-021 2.001 ) 


The  purpose  of  this  memorandum  is  to  provide  comments  on  selected  portions  of 
the  subject  draft  report. 

Recommendation  3.b.  Reassess  the  risk  of  exchanging  encrypted  files 
containing  sensitive  data  between  the  Reserve  Component  Automated  System  for 
proposed  and  fielded  increments  and  other  networked  systems  and,  if  appropriate, 
implement  encryption  technology.  The  CIO/G-6  concurs  with  this  recommendation. 

With  regard  to  the  information  assurance  findings  cited  on  page  10  of  the  report  the 
RCAS  will  again  be  due  for  re-accreditation  in  November  2002.  The  CIO/G-6  will 
continue  to  serve  as  the  Designated  Approving  Authority  (DAA)  for  the  RCAS.  During 
the  required  reviews,  prior  to  certification  and  accreditation,  the  PMO  and  the  RCAS 
Certification  Agent  will  reassess  the  risk  with  respect  to  passing  unclassified  information 
in  an  unencrypted  mode.  A  determination  will  be  made  if  the  data  and  system  users  are 
within  bounds  of  acceptable  risk.  The  DAA  will  review  and  determine  if  the  earlier 
deferment  for  deploying  data  encryption  hardware  should  be  allowed  to  continue. 
Alternatively  it  maybe  determined  that  additional  measures  should  be  put  in  place  to 
reduce  the  risk  of  inadvertent  disclosure  of  sensitive  but  unclassified  data. 

Recommendation  4.  b.  Recommend  that  the  Project  Management  Officer, 
Reserve  Component  Automation  System,  develop  a  plan  for  the  approval  of  the  Chief 
Information  Officer,  Department  of  the  Army  that  funds  the  operation  and  support  of  the 
Reserve  Component  Automation  System  for  FYs  2002  through  2007.  CIO/G-6  concurs 
with  this  recommendation.  With  regard  to  the  program  funding  findings  cited  on  page 
13  of  the  report  the  Army  Programs,  Analysis  and  Evaluation  (PA&E),  the  RCAS 
Program  Management  Office  (PMO),  the  National  Guard  Bureau  (NGB),  and  the  Office 
of  the  Chief,  Army  Reserve  (OCAR)  did  work  together  following  the  Increment  4/5, 
Milestone  llld,  review.  The  purpose  of  that  effort  was  to  resolve  how  to  support  the 
unfunded  requirements  for  operation  and  maintenance  of  the  RCAS  infrastructure 
during  the  period  FY02-07.  As  a  result  of  that  effort  it  was  determined  that  Increment  6, 
Milestone  llle,  was  affordable  as  documented  in  the  Army  PA&E  memorandum  dated 
January  21 , 2002  (enclosure).  Tabs  A  &  B  of  the  enclosed  PA&E  memorandum  are 
memoranda  from  the  OCAR  and  NGB  that  provide  their  commitments  to  support  the 
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operations  and  maintenance  of  the  RCAS  system.  The  Acquisition  Decision 
Memorandum  for  Milestone  I  lie,  therefore,  reflected  the  fact  that  the  increment  was 
ready  for  fielding.  The  Office  of  Secretary  of  Defense  (OSD)  Comptroller  and  Army 
Comptroller  both  concurred  with  the  proposed  resolution.  In  preparation  for  the 
Increment  6,  Milestone  llle,  the  Army  Chief  Information  Officer  (CIO)  performed  a  full 
review  of  the  RCAS  efforts  to  prepare  software  Increment  6  for  fielding,  and  found  that 
there  was  no  life  cycle  funding  issues.  Based  on  this  review,  the  CIO  Certification 
memorandum  was  signed  by  the  Army  CIO  on  February  25,  2002  and  forwarded  to  the 
Assistant  Secretary  of  Defense  for  Command,  Control,  Communications,  and 
Intelligence  (ASD(C3I))  who  is  also  the  DoD  CIO  and  the  RCAS  Milestone  Decision 
Authority  (MDA). 

The  Army  CIO  provided  a  concurrence  signature  for  the  Increment  6,  Milestone 
llle,  Acquisition  Decision  Memorandum  (ADM)  on  March  14,  2002.  The  RCAS  MDA, 

Mr.  Stenbit,  signed  the  Increment  6,  Milestone  llle,  ADM  on  March  25,  2002.  Mr. 
Stenbit's  signature  should  close  the  Army  and  RCAS  PEO  action  associated  with 
resolution  of  the  Block  4/5,  Milestone  1 1  Id,  unfunded  requirements  for  FY02-07  as 
reflected  in  the  Audit  Report. 

Questions  concerning  this  memo  should  be  addressed  to  Ms.  Jacqueline  Zachgo 
at  e-mail  Jacqueline.Zachao@hqda.armv.mil  or  by  phone  at  (703)  614-6167. 
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REPLY  TO 
ATTENTION  OF 


DEPARTMENT  OF  THE  ARMY 
OFFICE  OF  THE  DEPUTY  CHIEF  OF  STAFF  FOR  PROGRAMS 
700  ARMY  PENTAGON 
WASHINGTON.  DC  20310-0700 


January  21,  2002 


MEMORANDUM  FOR  PROGRAM  EXECUTIVE  OFFICER,  INFORMATION 
SYSTEMS 

SUBJECT:  Affordability  Assessment  for  the  Reserve  Component  Automation 
System  (RCAS) 


RCAS  is  an  AC  AT  1AM  program  that  has  had  Milestone  1 1  Id  approved  pending 
Clinger-Cohen  Act  (CCA)  certification.  On  July  2,  2001,  the  ASD  C3I  directed  the  Army 
CIO  and  PM  RCAS  to  conduct  an  affordability  analysis  in  coordination  with  the  Army 
National  Guard  and  the  US  Army  Reserve  as  a  function  of  the  CCA.  G8,  PA&E  has 
been  heavily  involved  in  the  reviews  of  RCAS  and  was  a  member  of  the  Integrating 
Integrated  Product  Team  Review  of  the  program.  What  follows  are  the  results  of  a 
series  of  reviews  begun  in  July  2001  and  completed  January  9,  2002. 

During  the  Army  Program  Objective  Memorandum  (POM)  03-07  build,  funding  for 
RCAS  was  increased  $410.6M,  or  44%,  over  the  Future  Years  Defense  Plan  to 
primarily  procure  lifecycle  replacement  hardware  and  additional  increments  of  software. 
Based  upon  this  fact,  the  Army  has  made  a  commitment  to  ensure  that  RCAS  remains  a 
viable  program  for  the  Reserve  Component  for  the  foreseeable  future. 

This  table  cites  the  requirements  profile  by  appropriation  provided  by  CEAC  from  the 
Army  Cost  Position  (ACP).  It  also  cites  the  funding  profile  by  appropriation  from  the 
BESPOM  03-07  file  from  the  Army’s  PROBE  database.  It  then  compares  the  fundina 
profile  and  the  ACP. 
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SUBJECT:  Affordability  Assessment  for  the  Reserve  Component  Automation 
System  (RCAS) 

a.  Although  OMAR  has  a  shortfall  of  $5.5M  in  FY02,  the  Army  Reserve  has 
provided  a  written  commitment  (at  TAB  A)  ensuring  that  shortfalls  will  be  funded.  Other 
funding  shortfalls  will  be  addressed  in  the  04-09  POM  build.  In  aggregate,  fuhdlng 
exceeds  requirements  by  $40.3M  for  the  OMAR  appropriation. 

b.  In  OMNG  there  is  a  shortfall  of  $7.9M  in  FY02.  The  National  Guard  Bureau 
has  provided  a  letter  (at  TAB  B)  indicating  that  by  having  the  States  make  tradeoffs  the 
shortfalls  can  be  paid.  Other  funding  shortfalls  will  be  addressed  in  the  POM  04-09 
build.  Overall  the  OMNG  funding  exceeds  requirements  by  $67M. 

c.  With  the  recently  applied  Congressional  mark  up  of  $15.5M  in  OPA,  the  UFR 
in  FY02  becomes  a  surplus  of  $7M.  There  is  a  $7M  shortfall  In  FY03  that  will  be 
addressed  In  the  POM  04-09  build.  With  the  Congressional  mark  applied  funding 
exceeds  requirements  from  FY04  to  07  by  $37.8M. 

d.  The  program's  overall  funding  exceeds  requirements  by  $145.1  M  with  the 
mark  applied. 

In  view  of  these  facts,  and  the  written  commitment  from  the  Components  ensuring 
this  program  will  be  successful,  I  view  this  program  as  affordable. 


Donald  C.  Tison 
Deputy  Director,  Program 
Analysis  and  Evaluation 


Enclosures 
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DEPARTMENT  OF  THE  ARMY 

OFRC6  OF  THE  CHIEF,  ARMY  RESERVE 
WASHINGTON.  DC  20310-2400 


1  g  DEC  1001 


MEMORANDUM  FOR  MG  Albert  J.  Madora,  Director,  Army  Program  Analysis 
and  Evaluation,  Washington,  DC  20310  (Room  3C718) 

SUBJECT:  Reserve  Component  Automation  System  (RCAS)  Affordability 

1 .  The  Army  Reserve  presented  the  RCAS  Life  Cycle  Support  (LCS) 
requirements  to  the  II  PEG  during  both  POM  02-07  and  03-07,  First  priority  was 
given  to  replacement  of  aging  hardware  components  and  second  priority  was 
operational  maintenance  support,  POM  03-07  fully  funded  the  replacement  of 
hardware  components  in  FY03,  but  due  to  Army  prioritization,  no  OMAR  funding 
was  allocated  for  RCAS  LCS  operational  and  maintenance  support  until  FY  06. 

2.  RCAS  provides  approximately  60%  of  the  Army  Reserve’s  IT  infrastructure 
and  backbone.  Validated  LCS  requirements  include  the  replacement  of  4,800 
obsolete  workstations  and  aging  backbone  hardware.  RCAS  enables  the  Army 
Resen/e  to  rapidly  mobilize  and  is  crucial  for  the  Army  Reserve’s  day-to-day 
operations. 

3.  Deferring  hardware  purchases  to  FY  03-07  eliminates  the  procurement 
shortfall  and  reduces  the  FY  02  funding  shortfall  to  approximately  $10.9M,  of 
which  $4.5M  is  the  Army  Reserve’s  price  to  support  system  operations.  We  have 
limited  flexibility  in  FY02  to  migrate  funding  to  cover  critical  shortfalls.  Un¬ 
resourced  requirements  are  being  compiled  for  consideration  at  the  appropriation 
level.  They  will  be  validated,  prioritized  and  submitted  to  the  Chief,  Army 
Reserve  for  funding  decisions.  We  assure  you  that  the  Army  Reserve  will  not 
allow  the  RCAS  program  to  fail,  sufficient  resources  will  be  made  available  to 
continue  this  critical  information  technology  program. 

FOR  THE  CHIEF,  ARMY  RESERVE: 


MALCOLM  B.  WESTCOTT 
COL,  GS 

Deputy  Chief,  Army  Reserve 
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DEPARTMENTS  op  the  army  and  the  air  FORCE 

NATIONAL  GUARD 'BUREAU 
141 1  JEFFERSON  DAVIS  HIGHWAY 
ARLINGTON.  VA  22202-3231 


21  November  2001 


memorandum  for  director,  programs  analysis  and  evaluation, 

WASHINGTON,  DC  20310 

SUBJECT:  Reserve  Component  Automation  System  (RCAS)  Affordability 

1 .  The  Fiscal  Year  (FY)  02-FY  07  POM  validated  and  prioritized  RCAS  operating  and  support 
requirements.  First  priority  was  given  to  replacement  of  aging  hardware  components  and 
second  priorities  were  the  operational  costs  and  maintenance.  Funding  limitations  prevented 
full  funding  for  either  replacement  or  Q&M  costs.  The  POM  FY  03-07  fully  funded  replacement 
hardware  components  starting  in  FY  03  and  the  operating  and  support  costs  in  FY  06  and  FY 
07.  A  backlog  of  hardware  replacement  remains  for  FY  00  through  FY  02,  but  this  backlog  is 
absorbed  over  the  course  of  the  FY  03-07  future  years  defense  program. 

2.  An  un-financed  requirement  of  approximately  $10M  per  year  for  operating  and  support  costs, 
predominantly  hardware  maintenance  remains  for  FY  02  through  FY  05.  This  shortfall  will  be 
funded  from  excess  procurement  as  a  result  qf  the  most  recent  Cost  and  Economic  Analysis 
Center  cost  estimate.  Therefore  no  additional  funding  is  necessary  for  FY  03-07,  Deferring 
hardware  purchases  to  FY  03-07  eliminates  the  procurement  shortfall  and  reduces  the  FY  02 
funding  shortfall  to  approximately  $10.9M,  $6.4M  for  the  Army  National  Guard  (ARNG)  to 
support  system  operations.  This  is  a  relatively  small  amount  of  our  total  information  technology 
maintenance  program. 

3.  The  ARNG  Is  committed  to  ensuring  continuation  of  RCAS  as  a  viable  capability  in  our 
overall  IT  infrastructure.  In  light  of  our  current  operational  situation,  there  is  a  pressing  need  to 
field  RCAS  increment  6  software  that  Supports  mobilization.  For  FY  02,  the  ARNG  has 
developed  a  funding  strategy  that  distributes  resources  to  the  States  rather  than  maintaining  a 
large  withhold  in  our  headquarters.  Un-financed  Requirements  are  sent  to  the  appropriate 
States  to  ensure  that  tradeoffs  are  made  with  readiness  and  mission  accomplishment  as  the 
primary  consideration.  As  such,  we  will  distribute  our  available  Director  of  Information 
Management  resources  and  allow  the  States  to  determine  prudent  tradeoffs  to  accomplish  the 
mission  and  maintain  our  information  technology  program.  The  ARNG  is  dependent  on  RCAS 
and  we  will  ensure  it  continues  to  function. 

4.  Point  of  contact  is  Ms.  Maureen  Wylie,  Division  Chief,  Program  Analysis  and  Evaluation,  at 
703-607-2898. 

FOR  THE  CHIEF,  NATIONAL  GUARD  BUREAU: 


NGB-ARA 
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Audit  Team  Members 

The  Acquisition  Management  Directorate,  Office  of  the  Assistant  Inspector  General  for 
Auditing  of  the  Department  of  Defense  prepared  this  report.  Personnel  of  the  Office  of 
the  Inspector  General  of  the  Department  of  Defense  who  contributed  to  the  report  are 
listed  below. 

Mary  Ugone 
Wanda  Scott 
James  Hutchinson 
Alvin  Lowe 
Jerry  Hall 
Christine  Winston 
Jacqueline  Pugh 


